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EXECUTIVE  SUMMARY 


1.0  Introduction 

This  executive  summary  serves  as  a  stand-alone  document  as  well  as  part  of  this  report.  For  that 
reason,  the  reader  will  find  some  duplication  of  verbiage  and  figures. 


2.0  JADS  Overview 

The  Joint  Advanced  Distributed  Simulation  (JADS)  Joint  Test  and  Evaluation  was  chartered  by 
the  Office  of  the  Under  Secretary  of  Defense  (Acquisition  and  Technology)  in  October  1994  to 
investigate  the  utility  of  advanced  distributed  simulation  (ADS)  technologies  for  support  of 
developmental  test  and  evaluation  (DT&E)  and  operational  test  and  evaluation  (OT&E).  The 
program  is  Air  Force  led  with  Army  and  Navy  participation.  The  program  is  currently  scheduled 
to  end  in  March  2000. 

The  JADS  Joint  Test  Force  is  directly  investigating  ADS  applications  in  three  slices  of  the  test 
and  evaluation  (T&E)  spectrum:  the  System  Integration  Test  (SIT)  explored  ADS  support  of  air- 
to-air  missile  testing;  the  End-to-End  (ETE)  Test  investigated  ADS  support  for  command, 
control,  communications,  computers,  intelligence,  surveillance,  and  reconnaissance  (C4ISR) 
testing;  and  the  Electronic  Warfare  (EW)  Test  explored  ADS  support  for  EW  testing.  Phase  4  of 
the  ETE  Test  is  the  subject  of  this  report. 

3.0  ETE  Test  Overview 

The  ETE  Test  was  designed  to  evaluate  the  utility  of  ADS  to  support  testing  of  C4ISR  systems. 
The  test  focused  on  the  Joint  Surveillance  Target  Attack  Radar  System  (Joint  STARS)  as  one 
component  of  a  representative  C4ISR  system.  The  ETE  Test  also  evaluated  the  capability  of  the 
JADS  Test  Control  and  Analysis  Center  (TCAC)  to  control  a  distributed  test  of  this  type  and 
remotely  monitor  and  analyze  test  results. 

The  ETE  Test  consisted  of  four  phases.  Phase  1  developed  or  modified  the  components  needed 
to  develop  the  ADS  test  environment.  Phase  2  used  the  ADS  test  environment  to  evaluate  the 
utility  of  ADS  to  support  DT&E  and  early  OT&E  of  a  C4ISR  system  in  a  laboratory 
environment.  Phase  3  transitioned  portions  of  the  architecture  to  the  E-8C  aircraft,  ensured  that 
the  components  functioned  properly,  and  checked  that  the  synthetic  environment  interacted 
properly  with  the  aircraft  and  actual  light  ground  station  module  (LGSM).  Phase  4  evaluated  the 
ability  to  perform  testing  and  evaluation  in  a  ADS -enhanced  live  test  environment. 
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4.0  Overview  of  ETE  Test  Phase  4 


4.1  Purpose 

Phase  4  determined  the  utility  of  ADS  in  performing  test  and  evaluation  in  an  ADS-enhanced 
live  test  environment.  The  test  objectives  were 

JADS  Issue  1.  What  is  the  present  utility  of  ADS,  including  distributed  interactive  simulation 
(DIS),  for  T&E? 

JADS  Objective  1-1.  Assess  the  validity  of  data  from  tests  utilizing  ADS,  including  DIS, 
during  test  execution. 

JADS  Objective  1-2.  Assess  the  benefits  of  using  ADS,  including  DIS,  in  T&E. 

JADS  Issue  2.  What  are  the  critical  constraints,  concerns,  and  methodologies  when  using  ADS 
for  T&E? 


JADS  Objective  2-1.  Assess  the  critical  constraints  and  concerns  in  ADS  performance  for 
T&E. 


JADS  Objective  2-2.  Assess  the  critical  constraints  and  concerns  in  ADS  support  systems 
for  T&E. 


JADS  Objective  2-3.  Develop  and  assess  methodologies  associated  with  ADS  for  T&E. 

4.2  Approach 


Figure  ES-1  provides  an  overview  of  the  Phase  4  ETE  Test  synthetic  environment. 
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A/C  =  aircraft  AFATDS  =  Advanced  Field  Artillery  Tactical  Data  System 

ATACMS  :=  Army  Tactical  Missile  System  Bn  =  battalion 

Janus  =  interactive,  computer-based  simulation  of  combat  operations  SCDL  =  surveillance  control  data  link 

T-1  =  digital  carrier  used  to  transmit  a  formatted  digital  signal  at  1 .544  megabits  per  second 

TAC  =  target  analysis  cell  TRAC  -  U.S.  Army  Training  and  Doctrine  Command  Analysis  Center 

VSTARS  =  Virtual  Surveillance  Target  Attack  Radar  System  WSMR  =  White  Sands  Missile  Range,  New  Mexico 


Figure  ES-1.  EXE  Test  Phase  4  Synthetic  Environment 


The  ETE  Test  used  the  Janus  6.88D  simulation  to  generate  the  entities  representing  the  elements 
in  the  rear  of  a  threat  force.  The  U.S.  Army  Training  and  Doctrine  Command  Analysis  Center 
(TRAC)  at  White  Sands  Missile  Range  (WSMR),  New  Mexico,  provided  the  Janus  scenario 
feed. 


The  Test  Control  and  Analysis  Center  in  Albuquerque,  New  Mexico,  provided  test  control. 

The  Joint  STARS  E-8C  simulation,  called  the  Virtual  Surveillance  Target  Attack  Radar  System 
(VSTARS),  represented  the  radar  subsystem  of  the  Joint  STARS  E-8C  when  the  test  was 
operating  in  a  laboratory  environment.  It  consisted  of  a  distributed  interactive  simulation 
network  interface  unit,  a  radar  processor  simulator  and  integrator  (RPSI)  that  contained  the  two 
real-time  radar  simulations  with  necessary  databases,  and  various  simulations  of  E-8C  processes. 
VSTARS  was  operated  at  the  Northrop  Grununan  Surveillance  and  Battle  Management  Systems 
facility  in  Melbourne,  Florida. 

When  the  test  was  conducted  in  a  live  mode,  an  ADS-enhanced  E-8C  provided  live,  virtual,  and 
mixed  radar  reports.  The  ADS-enhanced  E-8C  consisted  of  the  T3  aircraft  with  the  RPSI  and  the 
air  network  interface  unit  added  as  described  in  the  Phase  3  report. 


The  LGSM  and  target  analysis  cell  (TAC)  were  represented  by  Bravo  Company,  303d  Military 
hitelligence  Battalion.  Fire  support,  in  the  form  of  the  Advanced  Field  Artillery  Tactical  Data 
System  (AFATDS),  was  represented  by  soldiers  from  the  4th  Infantry  Division  (Mechanized). 
Communications  among  these  command,  control,  communications,  computers  and  intelligence 
(C4I)  systems  employed  doctrinally  correct  means. 

The  Tactical  Army  Fire  Support  Model  (TAFSM)  simulation  modeled  the  Army  Tactical  Missile 
System  (ATACMS)  battalion  and  sent  the  fire  and  detonate  protocol  data  units  (PDUs)  to  the 
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Janus  6.88D  simulation.  Janus  then  modeled  the  engagement  results  and  reflected  them  in  the 
synthetic  environment. 

5.0  ETE  Test  Phase  4  Results 

5.1  Fulfillment  of  Test  Objectives 

All  ETE  Test  Phase  4  objectives  were  met.  The  ETE  Test  team  determined  that  the  use  of  an 
ADS-enhanced  live  test  environment  strengthened  the  operational  testing  of  C4ISR  systems  of 
systems  and  resulted  in  the  collection  of  valid  data.  i 

During  Phase  4,  the  critical  constraints,  concerns,  and  methodologies  associated  with  using  an 
ADS-enhanced  test  environment  for  test  and  evaluation  were  also  evaluated. 

The  ETE  Test  team  also  investigated,  for  the  first  time,  the  use  of  satellite  communications  to 
transmit  simulation  data  to  an  aircraft  in  flight,  thereby  extending  ADS-enhanced  testing  into  a 
new  dimension. 

Finally,  the  ETE  Test  team  developed  and  assessed  test  control  and  data  collection 
methodologies  useful  for  ADS  testing. 

6.0  Lessons  Learned 

6.1  Technical 

Testers  should  carefully  plan  the  development  of  the  simulations  and  links  comprising  their  ADS 
environment.  During  test  execution,  they  must  ensure  that  the  time  sources  are  synchronized  and 
continuously  monitor  PDU  traffic.  The  distributed  nature  of  ADS  testing  necessitates  special 
equipment  for  network  check-out  and  verification  and  requires  strict  configuration  control  of 
analysis  tools  and  collected  data. 

6.2  Infrastructure  and  Process 

ADS  test  planning  should  be  detailed  enough  to  encompass  key  requirements  at  the  earliest 
possible  stages,  yet  flexible  enough  to  accommodate  unexpected  situations  during  test  execution. 
A  conservative  development  approach  is  recommended  —  accomplish  risk  reduction  activities 
before  each  ADS  test  and  let  each  ADS  test  build  on  the  success  of  earlier  experiments. 
Successful  test  execution  requires  effective  intemode  communication,  test  and  resource  control, 
and  data  management  procedures. 
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7.0  Conclusions 

7.1  Utility 

An  ADS-enhanced  live  environment  can  strengthen  OT&E  of  C4ISR  systems.  In  comparison 
with  conventional  tests,  an  ADS-enhanced  test  allows  the  examination  of  C4ISR  systems  under 
realistic  conditions  for  longer  periods  of  time,  over  far  larger  battlespaces,  and  at  a  much  lower 
cost.  This  versatile  technology  can  provide  test  environments  that  include  large  numbers  of 
entities,  entities  operating  under  realistic  but  unsafe  conditions  and  joint  and  combined 
operations.  ADS  provides  C4ISR  system  testers  with  greater  flexibility  in  designing,  executing, 
and  analyzing  their  tests. 

7.2  Technical 

Phase  4  used  both  a  conventional  wide  area  network  (WAN)  and  satellite  communications  to 
distribute  data.  The  WAN  exhibited  low  bandwidth  usage  and  a  low  latency  during  the  test.  The 
satellite  communications  used  all  the  available  bandwidth  and  required  buffering  to  handle 
periods  of  heavy  scenario  activity.  The  combination  of  buffering  and  transmission  times 
produced  relatively  high  latencies.  The  satellite  communications  also  had  a  higher  rate  of  data 
dropout  than  the  WAN.  All  these  effects  had  minimal  impact  on  the  test  but  should  be 
considered  when  testing  other  C4ISR  systems  using  satellite  communications. 

7.3  Infrastructure 

Compared  to  conventional  testing,  ADS  testing  reduces  the  need  for  large  numbers  of  fielded 
personnel  and  vehicles.  The  ability  to  automatically  collect  and  analyze  test  data  also  reduces  the 
number  of  people  required  for  setup,  execution,  and  analysis.  ADS  test  success  relies  on  well- 
organized  test  control  and  data  management  procedures.  Distributed  testing  requires 
sophisticated  instrumentation,  trained  personnel  to  operate  and  maintain  that  equipment,  and 
funds  to  support  personnel  and  equipment  at  distant  test  nodes. 
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1.0  Introduction 


1.1  Joint  Advanced  Distributed  Simulation  Overview 

The  Joint  Advanced  Distributed  Simulation  (IADS)  Joint  Test  and  Evaluation  (JT&E)  was 
chartered  by  the  Deputy  Director,  Test,  Systems  Engineering  and  Evaluation  (Test  and 
Evaluation)*,  Office  of  the  Under  Secretary  of  Defense  (OSD)  (Acquisition  and  Technology)  in 
October  1994  to  investigate  the  utility  of  advanced  distributed  simulation  (ADS)  technologies  for 
support  of  developmental  test  and  evaluation  (DT&E)  and  operational  test  and  evaluation 
(OT&E).  The  program  is  Air  Force  led  with  Army  and  Navy  participation.  Science  Applications 
International  Corporation  (SAIC)  and  the  Georgia  Tech  Research  Institute  provide  contracted 
technical  support.  The  program  is  currently  scheduled  to  end  in  March  2000. 

The  JADS  JT&E  charter  focuses  on  three  issues:  what  is  the  present  utility  of  ADS,  including 
distributed  interactive  simulation  (DIS),  for  test  and  evaluation  (T&E);  what  are  the  critical 
constraints,  concerns,  and  methodologies  when  using  ADS  for  T&E;  and  what  are  the 
requirements  that  must  be  introduced  into  ADS  systems  if  they  are  to  support  a  more  complete 
T&E  capability  in  the  future.  From  these  issues,  objectives  and  measures  have  been  developed  to 
guide  the  evaluation. 

The  JADS  Joint  Test  Force  (JTF)  is  directly  investigating  ADS  applications  in  three  slices  of  the 
T&E  spectrum:  the  System  Integration  Test  (SIT)  explored  ADS  support  of  air-to-air  missile 
testing;  the  End-to-End  (ETE)  Test  investigated  ADS  support  for  command,  control, 
conununications,  computers,  and  intelligence,  surveillance  and  reconnaissance  (C4ISR)  testing; 
and  the  Electronic  Warfare  (EW)  Test  explored  ADS  support  for  EW  testing.  Each  test  applied 
the  JADS  objectives  and  measures  as  appropriate  to  conduct  its  evaluation.  The  JTF  was  also 
chartered  to  observe  or  participate  at  a  modest  level  in  ADS  activities  sponsored  and  conducted 
by  other  agencies  in  an  effort  to  broaden  conclusions  developed  in  the  three  dedicated  test  areas. 

Phase  4  of  the  JADS  ETE  Test  is  the  subject  of  this  report  and  is  described  in  the  next  section; 
the  following  is  a  brief  synopsis  of  the  SIT  and  the  EW  Test. 

The  SIT  evaluated  the  utility  of  using  ADS  to  support  cost-effective  testing  of  an  integrated 
missile  weapon/launch  aircraft  system  in  an  operationally  realistic  scenario.  The  SIT  also 
evaluated  the  capability  of  the  JADS  Test  Control  and  Analysis  Center  (TCAC)  to  control  a 
distributed  test  of  this  type  and  to  remotely  monitor  and  analyze  test  results. 

More  detailed  information  on  the  SIT  can  be  found  in  the  test  reports  available  at 
http://www.jads.abq.com.  (After  1  March  2000  refer  requests  to  Headquarters  Air  Force 
Operational  Test  and  Evaluation  Center  (HQ  AFOTEC)/History  Office  (HO),  8500  Gibson  Blvd. 
SE,  JQrtland  Air  Force  Base,  New  Mexico  87117-5558  or  SAIC  Technical  Library,  2001  North 
Beauregard  St.  Suite  80,  Alexandria,  Virginia  22311.) 


'  This  office  is  now  the  Deputy  Director,  Developmental  Test  and  Evaluation  (DD,  DT&E). 
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The  EW  Test  evaluated  the  utility  of  ADS  in  a  distributed  EW  environment.  The  first  phase  was 
open  air  testing  to  develop  a  performance  baseline  for  two  subsequent  test  phases.  The  first 
distributed  test  phase  employed  a  linked  architecture  using  Department  of  Defense’s  (DoD)  high 
level  architecture  (HLA^  which  included  a  digital  simulation  model  of  the  ALQ-131  self¬ 
protection  jammer,  threat  simulation  facilities,  and  constructive  models  that  support  replication 
of  the  open  air  environment.  In  the  second  distributed  phase,  a  live  aircraft  and  jammer  in  an 
installed  systems  test  facility  was  substituted  for  the  digital  model,  hi  both  distributed  test 
architectures,  system  performance  data  were  compared  with  live  fly  data  for  verification  and 
validation  (V&V). 

More  detailed  information  on  the  EW  Test  can  be  found  in  the  test  reports  available  at 
http://www.jads.abq.com.  (After  1  March  2000  refer  requests  to  HQ  AFOTEC/HO,  8500  Gibson 
Blvd.  SE,  Kirtland  Air  Force  Base,  New  Mexico  87117-5558  or  SAIC  Technical  Library,  2001 
North  Beauregard  St.  Suite  80,  Alexandria,  Virginia  22311.) 

1.2  Test  Overview 

The  ETE  Test  was  designed  to  evaluate  the  utility  of  ADS  to  support  testing  of  C4ISR  systems. 
The  test  focused  on  the  Joint  Surveillance  Target  Attack  Radar  System  (Joint  STARS)  as  one 
component  of  a  representative  C4ISR  system.  The  ETE  Test  also  evaluated  the  capability  of  the 
JADS  TCAC  to  control  a  distributed  test  of  this  type  and  to  remotely  monitor  and  analyze  test 
results. 

The  ETE  Test  used  ADS  to  assemble  an  enhanced  environment  for  testing  C4ISR  systems.  The 
intent  was  to  provide  a  complete,  robust  set  of  interfaces  from  sensor  to  weapon  system, 
including  the  additional  intermediate  nodes  that  would  be  found  in  a  tactical  engagement.  The 
test  traced  a  thread  of  the  complete  battlefield  process  from  target  detection  to  target  assignment 
and  engagement  at  corps  level  using  ADS.  It  allowed  the  tester  to  evaluate  the  thread  as  a  whole 
or  the  contribution  of  any  parts  individually  and  to  evaluate  what  effects  an  operationally  realistic 
environment  had  on  the  system  under  test  (SUT). 

The  primary  method  used  by  the  ETE  Test  to  create  an  ADS-enhanced  test  environment  was  to 
seamlessly  add  additional  entities  to  the  battlefield  seen  by  Joint  STARS.  In  addition,  some  of 
the  complementary  suite  of  other  command,  control,  communications,  computers  and 
intelligence  (C4I)  and  weapon  systems,  with  which  Joint  STARS  would  interact,  were 
incorporated  into  the  environment  to  create  the  target  detection  to  target  engagement  loop.  The 
additional  entities  and  battlefield  elements  enabled  the  test  team  to  evaluate  the  utility  of  an 
ADS-enhanced  test  environment. 

The  test  concept  (Figure  1)  was  to  use  the  additional  entities  to  supplement  the  operational 
environment  experienced  by  the  E-8C  and  light  ground  station  module  (LGSM)  operators.  By 
mixing  available  live  targets  with  targets  generated  by  a  constructive  model,  a  battle  array 
approximating  the  major  systems  present  in  a  notional  corps  area  of  interest  was  presented. 
Construction  of  a  network  with  nodes  representing  appropriate  C4I  and  weapon  systems 
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presented  a  more  robust  cross  section  of  players  for  interaction  with  the  E-8C  and  LGSM 
operators. 


E-8C 


Test  Control 
&  Analysis 


SCDL 


Ground^ 

Statio: 


Virtual 

ATACMS 


FDC 


Constructivi 

Simulatio: 


TAG 


ATACMS  =  Army  Tactical  Missile  System  FDC  =  fire  direction  center  SCDL  =  surveillance  control  data  link 

TAG  =  target  analysis  cell 


Figure  1.  ETE  Test  Conceptual  Model 

Several  components  were  required  to  create  the  ADS-enhanced  operational  environment  used  in 
the  ETE  Test.  In  addition  to  Joint  STARS,  the  ETE  Test  required  a  validated  simulation  capable 
of  generating  entities  representing  the  rear  elements  of  a  threat  force.  The  ETE  Test  team 
modified  an  Army  verified  and  validated  simulation,  Janus,  to  meet  this  requirement.  Also,  a 
simulation  of  the  Joint  STARS  radar  was  needed  to  insert  the  simulated  entities  into  the  radar 
stream  on  board  the  E-8C  while  it  was  flying  a  live  mission.  Other  capabilities  used  to  support 
the  test  included  simulations  or  components  of  the  Army’s  artillery  command  and  control  process 
and  a  simulation  of  the  Army  Tactical  Missile  System  (ATACMS).  Communications  among 
these  simulations  were  accomplished  using  such  doctrinally  correct  means  as  the  CGS-lOO,  a 
subsystem  of  the  Compartmented  All  Source  Analysis  System  (ASAS)  Message  Processing 
System  (CAMPS),  remote  ASAS  workstations  (RWSs),  and  the  Advanced  Field  Artillery 
Tactical  Data  System  (AFATDS). 

The  ETE  Test  consisted  of  four  phases.  Phase  1  developed  or  modified  the  components  that 
allowed  the  mix  of  live  and  simulated  targets  at  an  E-8C  operator’s  console  and  LGSM 
operator’s  console.  Phase  2  verified  and  validated  the  simulations  used  in  the  ADS-enhanced 
environment  and  evaluated  the  utility  of  ADS  to  support  DT&E  and  early  OT&E  of  a  C4ISR 
system  in  a  laboratory  environment.  Phase  3  transitioned  portions  of  the  architecture  to  the  E-8C 
aircraft,  ensured  that  the  components  functioned  properly,  checked  that  the  synthetic  environment 
properly  interacted  with  the  adrcraft  and  the  actual  LGSM,  and  repeated  the  V&V.  Phase  4 
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evaluated  the  ability  to  conduct  operational  testing  using  an  ADS-enhanced  live  test 
environment. 

1.3  Phase  1  Overview 

During  Phase  1,  software  and  hardware  needed  to  establish  the  ETE  Test  ADS  environment  were 
developed,  modified,  and  integrated.  In  addition.  Phases  2  through  4  were  planned. 

The  ETE  Test  ADS  environment  components  developed  or  modified  during  Phase  1  included 
modification  of  a  constructive  simulation  (Janus)  to  provide  virtual  targets,  an  E-8C  radar 
simulation  called  the  Virtual  Surveillance  Target  Attack  Radar  System  (VSTARS),  an  interface 
to  allow  surveillance  control  data  link  (SCDL)  traffic  from  VSTARS  to  be  displayed  in  the 
LGSM,  and  an  initial  ADS  network  suitable  for  integration  and  testing. 

More  detailed  information  on  Phase  1  can  be  found  in  the  End-to-End  Interim  Report,  Phase  1, 
August  1998,  available  at  http://www.jads.abq.com.  (After  1  March  2000  refer  requests  to  HQ 
AFOTEC/HO,  8500  Gibson  Blvd.  SE,  Kirtland  Air  Force  Base,  New  Mexico  87117-5558  or 
SAIC  Technical  Library,  2001  North  Beauregard  St.  Suite  80,  Alexandria,  Virginia  22311.) 

1.3.1  Phase  1  Results 

Phase  1  identified  constraints  associated  with  ADS  testing.  One  key  constraint  was  the  ability  of 
the  DoD  modeling  and  simulation  infrastructure  to  support  ADS-enhanced  test  and  evaluation. 
A  measure  of  this  constraint  is  found  in  the  amount  of  effort  required  to  develop  the  simulations 
used  in  the  ADS-enhanced  test  environment.  Phase  1  illustrated  the  level  of  effort  and  associated 
costs  involved  in  developing  the  supporting  infrastructure  for  a  test  of  this  type. 

Phase  1  demonstrated  the  value  of  applying  a  systems  engineering  methodology  to  identify  the 
requirements  for  ADS  components,  evaluate  the  availability  of  ADS  components,  and  modify  or 
develop  the  components  to  meet  the  requirements. 

Phase  1  established  that  suitable  wide  area  networks  (WAN)  could  be  established  and  used  for 
ADS-enhanced  testing.  Extensive  testing  established  that  available  hardware  and  protocols  were 
able  to  handle  the  expected  load  and  that  network  latency  did  not  appear  to  be  a  problem.  After  a 
through  investigation  of  existing  logging  software,  none  of  which  proved  suitable,  it  was  decided 
to  develop  a  IADS  logger  and  player.  This  software  would  be  used  to  collect  test  data  and  to 
provide  post-test  playback  of  data  for  analysis,  troubleshooting  and  training. 

Data  analysis  tools  were  also  developed  to  aid  in  the  analysis  of  test  data.  These  tools  were 
incorporated  into  a  single  tool  called  the  JADS  Analysis  Toolbox.  Some  of  the  features  include 
protocol  data  unit  (PDU)  analysis  in  near  real  time,  predefined  analyses  and  outputs,  conversion 
of  binary  log  file  data  to  text  data,  PDU  replay,  and  conversion  utilities. 
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1.4  Phase  2  Overview 


Phase  2  of  the  ETE  Test  determined  the  utility  of  ADS  to  support  DT&E  and  early  OT&E  of  a 
C4ISR  system  in  a  laboratory  environment. 

In  Phase  2,  the  E-8C  aircraft  was  represented  by  a  laboratory-based  simulation,  VSTARS,  that 
emulated  the  E-8C  aircraft  and  provided  the  radar  reports  needed  to  drive  the  radar  displays  on 
the  Advanced  Technology  Work  Station  (ATWS)  and  the  LGSM.  The  Janus  model  simulated 
the  threat  battlefield  entities.  The  radar  reports  from  VSTARS  were  provided  to  an  actual  target 
analysis  cell  (TAG)  for  analysis  and  target  assignment.  Fire  support  missions  were  tasked  using 
AFATDS  messages  sent  to  the  Tactical  Army  Fire  Support  Model  (TAFSM).  TAFSM  then 
simulated  the  launch,  flyout,  and  detonation  of  an  ATACMS  missile.  Janus  calculated  the  results 
of  the  missile  impact  and  reported  them  to  VSTARS.  In  addition,  the  Janus  operator  could  alter 
the  scenario  to  reflect  combat  actions  taken  as  a  result  of  the  missile  strike.  These  data  were  also 
reported  to  VSTARS,  and  the  resulting  radar  reports  reflected  the  effects  of  the  missile  strike. 

More  detailed  information  on  Phase  2  can  be  found  in  the  End-to-End  Interim  Report,  Phase  2, 
February  1999  available  at  http://www.jads.abq.com.  (After  1  March  2000  refer  requests  to  HQ 
AFOTEC/HO,  8500  Gibson  Blvd.  SE,  Kirtland  Air  Force  Base,  New  Mexico.  87117-5558  or 
SAIC  Technical  Library,  2001  North  Beauregard  St.  Suite  80,  Alexandria,  Virginia  2231 1.) 

1.4.1  Phase  2  Results 

All  ETE  Test  Phase  2  objectives  were  met.  Phase  2  of  the  ETE  Test  illustrated  the  utility  of  ADS 
to  support  DT&E  and  early  OT&E  of  a  C4ISR  system  in  a  laboratory  environment.  Specific 
ETE  Test  Phase  2  results: 

-  An  ADS  environment  can  dramatically  enhance  C4ISR  system  DT&E  and  early  OT&E. 

-  ADS  testing  provides  the  ability  to  test  C4ISR  systems  of  systems  prior  to  the  development 
of  major  systems  that  make  up  the  systems  of  systems. 

-  The  Phase  2  test  required  only  a  small  part  of  the  available  WAN  bandwidth  and  exhibited  a 
low  PDU  latency  rate  comparable  with  earlier  tests. 

-  The  ETE  Test  WAN  was  highly  reliable  during  Phase  2  testing,  largely  because  of  the  ETE 
Test  team’s  extensive  pretest  risk  reduction  efforts. 

-  The  requirement  to  field  large  numbers  of  personnel  and  vehicles  to  test  complex  C4ISR 
systems  is  greatly  reduced  when  ADS  is  used  to  augment  the  battlefield. 

-  An  ADS  laboratory  environment  can  also  be  used  for  test  planning,  rehearsal,  and  execution. 

-  Valid  test  data  can  be  collected  using  an  ADS  laboratory  environment. 
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Phase  2,  also  identified  critical  constraints,  concerns,  and  methodologies  associated  with  using 
ADS  for  test  and  evaluation  and  developed  and  assessed  test  control  and  data  collection 
methodologies  useful  for  ADS  testing. 

1.5  Phase  3  Overview 

The  ETE  Test  Phase  3  provided  an  iterative  step  in  determining  the  utility  of  ADS  to  the  T&E  of 
a  C4ISR  system.  During  this  phase,  portions  of  the  architecture  were  transferred  to  the  E-8C 
aircraft,  the  components  were  checked  to  make  sure  that  they  functioned  properly,  and  the 
synthetic  environment  was  checked  to  make  sure  that  it  interacted  properly  with  the  aircraft  and 
actual  ground  station  module  (GSM). 

I 

Phase  3  focused  on  migrating  certain  software  components  of  VSTARS,  specifically  the  air 
network  interface  unit  (ANIU)  and  the  radar  processor  simulator  and  integrator  (RPSI),  from  the 
laboratory  Alpha  workstations  to  the  primary  mission  equipment  on  the  T3  E-8C  aircraft.  In 
addition,  the  ground  network  interface  unit  (GNIU)  software  was  separated  from  VSTARS  and 
migrated  to  an  Alpha  workstation  collocated  with  a  satellite  transceiver. 

More  detailed  information  on  Phase  3  can  be  found  in  the  End-to-End  Interim  Report,  Phase  3, 
May  1999  available  at  http://www.jads.abq.com.  (After  1  March  2000  refer  requests  to  HQ 
AFOTEC/HO,  8500  Gibson  Blvd.  SE,  Kirtland  Air  Force  Base,  New  Mexico  87117-5558  or 
SAIC  Technical  Library,  2001  North  Beauregard  St.  Suite  80,  Alexandria,  Virginia  22311.) 

1.5.1  Phase  3  Results 

The  ETE  Test  team  successfully  collected  performance  data  on  the  network  environment,  while  it 
operated  during  Phase  3.  In  addition,  the  ETE  Test  team  developed  and  assessed  test  control  and 
data  collection  methodologies  useful  for  ADS  testing. 

A  series  of  formal  system  integration  tests  (SIT)  of  the  Joint  STARS  aircraft  with  the  radar 
simulation  operating  was  performed  by  the  Joint  STARS  JTF. 

The  ETE  Test  team  also  conducted  a  formal  V&V  of  the  VSTARS  software  on  board  the  aircraft. 
The  testing  used  recorded  Janus  vignettes  played  from  equipment  located  in  the  Northrop 
Gmmman  laboratory,  then  broadcast  via  a  satellite  communications  (SATCOM)  link  to  the 
aircraft  located  on  the  tarmac.  A  GSM,  located  at  the  Joint  STARS  JTF  facility,  was  used  to 
verify  SCDL  linking  functions  with  the  aircraft.  All  the  formal  testing  conducted  by  the  Joint 
STARS  JTF  was  completed  successfully  with  only  minor  discrepancies  noted.  The  V&V  was 
conducted  concurrently  with  the  SIT. 
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2.0  Phase  4  Overview 


2.1  Phase  4  Purpose 

Phase  4  evaluated  the  ability  to  perform  test  and  evaluation  in  a  synthetically  enhanced 
operational  environment  using  typical  operators. 

2.2  Phase  4  Approach 

All  EXE  Test  Phase  4  objectives  were  met.  The  ETE  Test  team  determined  that  the  use  of  an 
ADS-enhanced  live  test  environment  strengthened  the  operational  testing  of  C4ISR  systems  of 
systems  and  resulted  in  the  collection  of  valid  data. 

Phase  4  determined  the  utility  of  ADS  in  augmenting  a  live,  open  air  test  mission  of  a  C4ISR 
system.  ADS  was  used,  as  in  Phase  2,  to  present  a  synthetic  environment  that  was  more 
representative  of  an  operational  theater  than  would  be  found  at  a  test  range.  The  Phase  4  test  was 
conducted  over  Fort  Hood,  Texas.  A  setup/rehearsal  period  was  followed  by  the  E-8C  flying 
three  test  missions.  The  VSTARS  in  the  Gramman  laboratory,  as  discussed  for  Phase  2,  was 
used  for  the  remaining  test  missions.  All  SUT  data  collected  from  these  live  test  missions  were 
compared  with  previous  test  mission  data  collected  during  Phase  2  and  the  Phase  4  laboratory 
test.  In  addition,  the  synthetic  environment  was  closely  monitored  to  collect  DIS  component 
performance  data,  to  assess  the  impact  of  DIS  component  performance  on  the  SUT  data,  and  to 
identify  problems  with  DIS  components  and  test  methodologies  that  impact  SUT  data  validity. 
These  results  were  used,  as  appropriate,  to  help  establish  requirements  for  subsequent  ADS 
technology  growth. 

Figure  2  shows  the  organizational  stmcture  for  reporting  and  coordination  during  Phase  4  of  the 
ETE  Test. 
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Bde  =  brigade 

D&SA  =  Depth  and  Simultaneous  Attack 
ID  =  infantry  division 
MI  =  Military  Intelligence 

TEXCOM  =  U.S.  Army  Test  and  Experimentation  Command 
WSMR  =  White  Sands  Missile  Range 


Bn  =  battalion  Co  =  company 

DDS A  =  deputy  director.  System  Assessment 
JPO  =  joint  program  office 
PM  =  program  manager 

TRAC  =  U.S.  Army  Training  and  Doctrine  Command  Analysis  Center 


Figure  2.  EXE  Test  Organizational  Structure 

During  Phase  4  testing,  the  roles  and  responsibilities  of  these  organizations  were  as  follows. 

Deputy  Director,  System  Assessment  (DDSA)* 

/ 

DDS  A,  Washington,  District  of  Columbia: 

•  Oversaw  the  JADS  Joint  Test  and  Evaluation  (JT&E) 

•  Approved  JADS  financial  requirements 

•  Approved  the  program  test  plan  (PTP) 

•  Oversaw  the  analysis  and  reporting  of  test  results 

JADS  JTF 

The  JADS  JTF,  Albuquerque,  New  Mexico: 

•  Conducted  overall  planning,  execution,  analysis,  and  reporting  of  the  test 

•  Managed  funding  to  accomplish  the  test 

•  Developed  and  evaluated  JADS  issues,  objectives,  measures,  and  related  data  elements 


^  This  office  is  now  the  Deputy  Director,  Developmental  Test  and  Evaluation  (DD,  DT&E). 
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•  Developed  and  integrated  the  components  of  the  ETE  Test  ADS-enhanced  live  test 
environment 

•  Established  necessary  communication  links  with  test  participants 

•  Operated  the  Test  Control  and  Analysis  Center  during  testing 

•  Worked  with  other  organizations  in  analyzing  test  data 

•  Reported  Interim  and  final  results  to  OSD 

U.S.  Army  Training  and  Doctrine  Command  Analysis  Center  (TRAC),  White  Sands 
Missile  Range  (WSMR) 

TRAC-WSMR,  New  Mexico: 

•  Developed,  tested,  and  documented  Janus  6.88D  (an  expanded  variant  of  Janus)  for  JADS 

•  Assisted  in  integrating  Janus  6.88D  into  the  ETE  Test  ADS-enhanced  live  test  environment 

•  Assisted  in  database  conversions 

•  Assisted  in  developing  vignettes 

•  Assisted  in  verification,  validation,  and  accreditation  (VV&A)  activities 

•  Assisted  in  ETE  Test  execution 

U.S.  Army  Test  and  Experimentation  Command  (TEXCOM)  Lab 
The  TEXCOM  lab.  Fort  Hood,  Texas: 

•  Assisted  in  scenario  and  vignette  development 

•  Assisted  in  ETE  Test  execution 

Depth  and  Simultaneous  Attack  (D&SA)  Battle  Lab 
The  D&SA  Battle  Lab,  Fort  Sill,  Oklahoma: 

•  Provided  and  operated  the  TAFSM  and  AFATDS 

•  Assisted  in  the  integration  of  the  ETE  Test  ADS-enhanced  live  test  environment 

•  Assisted  in  VV&A  activities  and  ETE  Test  execution 

U.S.  Army  III  Corps 

in  Corps  Headquarters,  Fort  Hood,  Texas: 

•  B  Company  (Co),  303d  Military  Intelligence  (MI)  Battalion  (Bn),  504  MI  Brigade  (Bde) 
supported  the  conduct  of  ETE  Test  Phase  2  and  Phase  4  events  with  LGSM(s)  and  a  target 
analysis  cell  (TAC)  and  assisted  in  the  integration  of  the  ETE  Test  ADS-enhanced  live  test 
environment 

•  504  MI  Bde  provided  a  test  environment  for  the  ETE  Test  Phases  2  and  4 

•  Fire  Support  4th  Infantry  Division  (4  ID)  provided  an  AFATDS  and  personnel  to  support  the 
ETE  Test  Phase  4 
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Joint  STARS  Joint  Program  Office  (JPO) 

Joint  STARS  JPO,  Hanscom  Air  Force  Base  (AFB),  Massachusetts,  provided  access  to  the  Joint 
STARS  JTF  and  Northrop  Grumman. 

The  Joint  STARS  JTF  of  the  Joint  STARS  JPO  in  Melbourne,  Florida: 

•  Supported  conduct  of  testing  in  all  phases 

•  Analyzed  Joint  STARS  test  results  and  provided  evaluations  according  to  JADS  objectives 

•  Assisted  in  VV&A  activities 

•  Provided  operators  during  Phase  4  of  the  ETE  Test 

Northrop  Grumman  Aerospace  Corporation 

Northrop  Grumman,  Electronics  and  Systems  Integration  Division,  Melbourne,  Florida: 

•  Designed,  developed  and  integrated  the  radar  processor  simulation  and  integrator 

•  Developed  the  Virtual  Surveillance  Target  Attack  Radar  System 

•  Conducted  and  assisted  in  V&V  activities 

•  Assisted  in  E-8C  mission  planning 

•  Operated  VSTARS  during  ETE  Test  phases 

Contracting  with  Northrop  Grumman  was  conducted  through  Rome  Laboratory  in  New  York. 

2.3  Test  Objectives 

Phase  4  determined  the  utility  of  ADS  in  performing  test  and  evaluation  in  a  synthetically 
enhanced  live  test  environment. 

The  JADS  issues,  test  objectives,  and  subobjectives  for  Phase  4  are  described  below.  Each 
subobjective  in  turn  encompassed  one  or  more  test  measures.  In  Section  4  these  issues, 
objectives,  subobjectives,  and  test  measures  are  discussed  in  terms  of  their  intent,  the  associated 
data  collection  methodology,  and  operational  test  results. 

JADS  Issue  1.  What  is  the  present  utility  of  ADS,  including  DIS,  for  T&E? 

JADS  Objective  1-1.  Assess  the  validity  of  data  from  tests  utilizing  ADS,  including  DIS, 
during  test  execution. 

JADS  Objective  1-2.  Assess  the  benefits  of  using  ADS,  including  DIS,  in  T&E.  This  test 
objective  was  broken  down  into  subobjectives. 

JADS  Subobjective  1-2-1.  Assess  ADS  capability  to  support  the  early  phases  of  the 
acquisition  process. 
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JADS  Subobjective  1-2-2.  Assess  ADS  capability  to  support  T&E  planning  and  test 
rehearsal. 

JADS  Subobjective  1-2-3.  Assess  ADS  capability  to  support  T&E  execution. 

JADS  Issue  2.  What  are  the  critical  constraints,  concerns,  and  methodologies  when  using  ADS 
for  T&E? 

JADS  Objective  2-1.  Assess  the  critical  constraints  and  concerns  in  ADS  performance  for 
T&E.  This  objective  was  broken  down  into  subobjectives.  (Subobjective  2-1-1  is  not 
applicable  to  the  ETE  Test  Phase  4.) 

JADS  Subobjective  2-1-2.  Assess  network  and  communications  performance 
constraints  and  concerns. 

JADS  Subobjective  2-1-3.  Assess  the  impact  of  ADS  reliability,  availability,  and 
maintainability  on  T&E. 

JADS  Objective  2-2.  Assess  the  critical  constraints  and  concerns  in  ADS  support  systems 
for  T&E.  This  objective  was  broken  down  into  subobjectives. 

JADS  Subobjective  2-2-1.  Assess  the  critical  constraints  and  concerns  regarding  ADS 
data  management  and  analysis  systems. 

JADS  Subobjective  2-2-2.  Assess  the  critical  constraints  and  concerns  regarding 
configuration  management  of  ADS  test  assets. 

JADS  Objective  2-3.  Develop  and  assess  methodologies  associated  with  ADS  for  T&E. 
(Subobjectives  2-3-1, 2-3-3,  and  2-3-4  are  not  applicable  to  the  ETE  Test  Phase  4.) 

JADS  Subobjective  2-3-2.  Develop  and  assess  methodologies  associated  with  test 
execution  and  control  for  tests  using  ADS. 

2.4  Phase  4  Methodology 

2.4.1  Tactical  Vignettes 

The  ETE  Test  team  enhanced  an  unclassified,  U.S.  Army  Training  and  Doctrine  Command 
(TRADOC)-approved,  54-hour  corps  battlefield  simulation  (CBS)  scenario  by  replicating  an 
Iraqi  corps  rear  area  of  operations  in  Iraq.  Table  1  describes  the  five  tactical  vignettes  created  in 
Janus  6.88D;  each  vignette  covered  a  six-hour  period  of  the  battle.  Representative  targets  present 
in  the  150  x  150  kilometer  (km)  Southwest  Asia  (SWA)  terrain  box  were  air  defense  artillery 
(ADA)  sites,  command  and  control  sites,  lines  of  communications  (convoys),  logistics  bases,  and 
concentrations  of  armor  and  artillery  units. 
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Table  1.  Vignettes  Used  During  ETE  Testing 


Vignette 

Description 

Number  of 
Entities 

1 

Prehostility  phase 

9,897 

2 

■  Preemptive  strikes 

9,757 

3 

Hammurabi  Division  logistical  operations 

9,904 

4 

Commitment  of  the  Hammurabi  Division 

9,781 

5 

General  headquarters  (GHQ)  depots  to 
corps  and  divisional  logistical  operations 

9,950 

2-4.2  Test  Configuration 

2.4.2. 1  Phase  4  Synthetic  Environment 

Several  components  were  required  to  create  the  ADS-enhanced  live  test  environment  used  in 
Phase  4.  Figure  3  provides  an  overview  of  the  Phase  4  synthetic  environment. 


A/C  =  aircraft 


T-I  =  digital  carrier  used  to  transmit  a  formatted  digital  signal  at  1.544  megabits  per  second 

Figures.  ETE  Test  Phase  4  Synthetic  Environment 


TRAC-WSMR  provided  the  scenario  feed  for  Phase  4  of  the  ETE  Test  using  Janus  6.88D  to 
generate  the  entities  representing  the  threat’s  rear  elements.  The  entities  were  sent,  using  entity, 
state  PDUs  (ESPDUs),  to  the  E-8C  via  the  Test  Control  and  Analysis  Center  (TCAC). 

The  TCAC  in  Albuquerque,  New  Mexico,  provided  test  control.  The  IADS  Network  and 
Engineering  (N&E)  team  monitored  the  health  of  the  ETE  Test  network  and  ensured  that 
adequate  data  flowed  in  support  of  the  test. 


The  Joint  STARS  E-8C  simulation,  VSTARS,  represented  the  Joint  STARS  E-8C  in  a  laboratory 
environment.  It  consisted  of  a  distributed  interactive  simulation  network  interface  unit  (NIU),  an 
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RPSI  that  contained  the  two  real-time  radar  simulations  with  necessary  databases,  and  various 
simulations  of  E-8C  processes.  Figure  4  provides  more  information  on  the  VSTARS 
architecture.  VSTARS  was  operated  at  the  Northrop  Grumman  Surveillance  and  Battle 
Management  Systems  facility  in  Melbourne,  Rorida. 


Dwell 

Target 


VSTARS  Data  ^ ^ 
Packet  ^  PTin 

RTareeMDa^S 

CDP  -  central  data  processor 

MTI  =  moving  target  indicator 

SM&C  -  system  management  and  control 

T-1  =  digital  carrier  used  to  transmit  a  formatted  digital 


M&IS  -  management  and  integration  software 
SAR  =  synthetic  aperture  radar 
SMO  -  system  management  officer 
signal  at  1.544  megabits  per  second 


Figure  4.  VSTARS  Architecture 

The  TAG,  fire  support  command  and  control  elements  (provided  by  the  AFATDS),  and  a  LGSM 
were  stationed  at  Fort  Hood,  Texas. 

When  the  test  was  conducted  in  a  live  mode,  an  ADS-enhanced  E-8C  provided  live,  virtual,  and 
mixed  radar  reports.  The  ADS-enhanced  E-8C  consisted  of  the  T3  aircraft  with  the  RPSI  and  the 
air  network  interface  unit  added  as  described  in  the  Phase  3  report.  Figure  5  provides  more 
information  on  the  ADS-enhanced  E-8C  configuration. 
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SE=  synthetic  environment  T-1  =  digital  carrier  used  to  transmit  a  formatted  digital  signal  at  1 .544  megabits  per  second 

Figure  5.  ADS-Enhanced  E-8C  Configuration 

Communications  among  these  C4I  systems  employed  such  doctrinally  correct  means  as  the  CGS- 
100,  a  subsystem  of  the  CAMPS,  remote  workstations,  and  AFATDS  message  traffic. 

The  AFATDS  messages  were  transmitted  between  the  AFATDS  located  at  Fort  Hood  and  the 
AFATDS  located  at  Fort  Sill  using  actual  tactical  protocols  rather  than  DIS  PDUs.  Also,  the 
SCDL  messages  were  transmitted  between  VSTARS  and  the  LGSM  using  a  dedicated  link,  a 
special-purpose  interface,  and  the  actual  tactical  protocols. 

The  LGSM  and  TAC  were  represented  by  Bravo  Company,  303d  Military  Intelligence  Battalion. 
The  AFATDS  was  manned  by  soldiers  from  the  4th  Infantry  Division  (Mechanized). 

The  TAFSM  simulation  modeled  the  ATACMS  battalion  and  sent  the  fire  and  detonate  PDUs  to 
the  Janus  6.88D  simulation.  Janus  then  modeled  the  engagement  results  and  reflected  them  in 
the  synthetic  environment  (SE). 
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2.4.22  Phase  4  Network 


Figures  6  and  7  provide  a  more  detailed  description  of  the  ETE  Test  network. 
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Figure  6.  ETE  Test  Phase  4  Network  Diagram  (Virtual  Phase) 
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Figure  7.  EXE  Test  Phase  4  Network  Diagram 


2.4.23  Test  Control  and  Monitoring 

During  Phase  4,  ETE  Test  and  N&E  team  members  performed  test  control  from  the  TCAC.  Test 
control  consisted  of  three  major  areas  —  network  monitoring,  communications,  and  test 
procedures. 

The  N&E  team  conducted  network  monitoring  in  the  TCAC  using  hardware  and  software  tools. 
The  software  consisted  of  commercial  products  and  test-specific  tools  developed  by  JADS 
analyst/programmers.  They  used  the  following  systems. 

Silicon  Graphics,  Inc.  (SGI)  Indy  -  JADS  logger 

SGI  Indy  -  time  server 

SGI  Indigo  -  NetVisualyzer™ 

SUN  SPARC  5  -  SPECTRUM® 

Line  printers 

JADS  analyst/programmers  developed  the  JADS  logger.  This  software  recorded  all  PDU  traffic 
at  individual  sites.  All  nodes,  with  the  exception  of  Fort  Hood,  had  a  JADS  logger  installed. 
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The  logger  recorded  the  receipt  of  the  PDU  and  time  stamped  it  using  an  accurate  time  source. 
These  data  were  used  to  analyze  PDU  transmission  performance  over  the  network. 

JADS  analyst/programmers  also  developed  a  time  server  that  provided  the  accurate  time  source 
needed  for  the  JADS  logger.  The  time  server  was  tied  to  a  global  positioning  system  (GPS) 
receiver  located  in  the  TCAC  and  provided  time  to  all  nodes  with  an  accuracy  of  100 
microseconds.  The  software  also  contained  monitoring  tools  to  track  the  time  servers 
performance  over  an  8-hour  test  period. 

Cabletron  SPECTRUM®,  NetVisualizer™,  and  line  printers  were  used  to  provide  network 
monitoring.  SPECTRUM®  measured  bandwidth  utilization.  This  tool  recorded  the  percentage 
of  bandwidth  used,  as  well  as  bandwidth  loading  on  a  network  segment.  NetVisualizer™ 
software  displayed  real-time  bandwidth  use  in  a  rolling  bar  graph  format  for  quick,  visual 
reference.  The  line  printers  provided  a  printout  of  network  router  status.  Any  failure  or  high  bit 
error  rate  resulted  in  a  printout  showing  the  problem  and  identity  of  the  offending  router. 

Communication  among  the  distributed  ETE  Test  nodes  was  critical.  The  TCAC  provided  all 
needed  communication  systems.  Dedicated  phone  circuits  residing  on  the  T-1  lines  provided 
classified  and  unclassified  service.  The  unclassified  line  allowed  connectivity  among  the  TCAC, 
Fort  Hood,  Fort  Sill,  and  TRAC-WSMR.  The  classified  line  allowed  connectivity  among  the 
TCAC,  Fort  Hood,  and  Northrop  Gmmman.  These  lines  allowed  the  operator  to  select  the 
desired  site,  lift  the  receiver,  and  connect  directly  to  that  site.  The  TCAC  could  select  multiple 
sites  for  conference  calls  on  these  lines.  In  addition  to  these  lines,  the  ETE  Test  team  used  an 
unclassified  conference  line  to  coordinate  such  test  events  as  network  checks  and  after-action 
debriefs.  This  line  allowed  up  to  ten  participants  to  connect  at  one  time. 

Test  procedures  were  required  to  provide  effective  control  of  all  test  nodes  during  test  events. 
The  test  procedures  were  in  checklist  format,  which  provided  for  standardization  among  the 
distributed  nodes.  The  network  checklist  was  most  critical  and  was  used  to  initialize  the  network 
before  the  test.  Other  checklists  included  those  used  to  start  up  hardware  and  software  at 
individual  nodes,  as  well  as  the  checklist  used  by  the  TCAC  test  controller  to  start  and  stop  the 
overall  test. 

2.5  Phase  4  Schedule 

Figure  8  provides  a  schedule  of  the  top-level  tasks  for  Phase  4  of  the  ETE  Test.  Phase  4  testing 
proceeded  as  scheduled. 
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Task  Name 

FY99,  Qtrl 

FY99,Qtr2 

FY99,  Qtr3 

FY  99,  Qtr  4 

Establish  Network 

Connectivity  Test 

Conduct  Test 

Quick-Look  Report 

Test  Report 

t 

t 

t 

1 

k 

t 

t  1 

Scheduled  completion  "^Actual  completion  o 


Previous  scheduled  Previous  scheduled 

completion  -  still  in  future^r  completion  -  date  passed 


Figures.  ETE  Test  Schedule 


2.6  Phase  4  Costs 

Appendix  D  of  this  report  is  a  work  breakdown  structure  covering  the  costs  of  all  four  phases  of 
the  ETE  Test. 
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3.0  Phase  4  Execution  Results 

3.1  Risk  Reduction  Tests 

3.1.1  15  March  Trial 

The  Northrop  Grumman  VSTARS  node  was  unable  to  establish  SCDL  during  the  15  March  trial 
because  it  was  attempting  to  use  the  SCDL  software  developed  for  the  aircraft.  This  software 
was  incompatible  with  the  laboratory  computer  workstations  used  during  the  trial.  A  tape  of  the 
radar  reports  derived  from  the  Janus  scenario  was  required  and  played  at  Fort  Hood;  the  rest  of 
the  nodes  received  the  live  Janus  feed.  Tape  playback  problems  occurred  at  Fort  Hood  resulting 
in  cancellation  of  the  trial.  The  SCDL  problems  at  Northrop  Grumman  were  resolved  by  using  a 
previous  version  of  the  SCDL  software  modified  to  run  under  the  Phase  4  test  conditions.  No 
nodes  were  required  to  run  from  tape  for  the  remainder  of  the  test. 

3.1.2  16  March  Trial 

A  successful  laboratory  trial  was  completed  on  16  March  with  no  major  problems  noted  at  any 
node. 


3.1.3  17  March  Trial 

VSTARS  problems  were  experienced  during  the  17  March  trial.  VSTARS  was  brought  down  to 
make  software  adjustments  to  keep  entities  on  roads.  This  was  required  because  an  earlier 
version  of  the  SCDL  software  was  used  as  discussed  above.  VSTARS  crashed  later  in  the  trial 
because  of  a  disk  usage  problem.  These  problems  were  resolved  before  the  end  of  the  trial,  and 
all  nodes  were  prepared  for  the  first  ETE  Test  live  flight  on  19  March. 

3.1.4 19  March  Live  Flight 

During  the  19  March  live  flight,  communication  problems  between  the  TCAC  and  the  E-8C 
resulted  in  the  desired  scenario  start  time  being  unclear.  Janus  and  TAFSM  were  started  too 
soon  and  were  stopped  until  the  radar  simulation  was  functioning  properly  on  board  the  E-8C. 
Once  the  radar  simulation  was  running,  the  simulations  were  restarted  and  start-up  data  were 
uploaded.  When  the  aircraft  arrived  on  station,  a  SCDL  uplink  could  not  initially  be  established. 
Numerous  corrective  measures  were  attempted  at  Fort  Hood  and  on  board  the  E-8C.  Following  a 
reboot  of  certain  SCDL  processes  on  the  E-8C,  the  SCDL  problem  was  resolved.  Problems  with 
SCDL,  along  with  intermittent  satellite  outages,  continued  throughout  the  flight.  Despite  the 
problems  with  SCDL  and  the  satellite  link,  the  19  March  live  flight  accomplished  all  the 
verification,  validation,  and  test  objectives  planned  for  the  flight.  The  19  March  flight  was  a 
successful  implementation  of  an  ADS-enhanced  live  test  environment  and  was  notable  for  four 
key  events. 
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1.  The  Joint  STARS  system  incorporating  the  JADS  capability  exhibited  normal  in-flight 
operation  without  any  noticeable  performance  degradation. 

2.  Scenario  data  were  received  from  White  Sands  via  SATCOM  and  displayed  in  near  real 
time  on  the  aircraft. 

3.  The  scenario  data,  as  reflected  in  the  radar  reports,  were  transmitted  through  the  SCDL  link 
to  the  GSM  at  Fort  Hood. 

4.  Radar  reports  that  contained  live,  virtual,  and  mixed  areas  within  the  ground  reference 
coverage  area  (GRCA)  were  generated  and  displayed  on  the  ATWSs  and  GSM. 

3.2  Operational  Testing 

3.2.1  24  March  Trial 

The  24  March  trial  was  a  laboratory  trial  that  encountered  difficulties  with  the  VSTARS 
synthetic  aperture  radar  (SAR)  simulation.  The  problem  was  a  reoccurrence  of  the  disk  problem 
that  occurred  during  the  17  March  trial.  The  equipment  was  repaired  before  the  next  laboratory 
trial,  and  caused  no  further  test  delays  or  stoppages.  The  problem  did  prevent  the  availability  of 
SARs  during  the  trial  causing  VSTARS  to  operate  in  a  degraded  moving  target  indicator  (MTI)- 
only  mode.  All  other  nodes  executed  the  24  March  trial  without  any  problems. 

3.2.2  25  March  Live  Flight 

For  the  25  March  live  flight,  excellent  communication  and  coordination  among  the  TCAC, 
Northrop  Grumman  lab,  and  the  E-8C  resulted  in  a  smooth  test  start-up  and  transition.  The  test 
mission  accomplished  the  majority  of  the  verification,  validation,  and  test  objectives  planned  for 
the  flight.  Because  of  E-8C  radar  problems  experienced  near  the  end  of  the  flight,  the  mixed 
radar  portion  of  the  test  could  not  be  evaluated.  Resolution  of  the  radar  problems  occupied  the 
remainder  of  the  available  station  time,  and  the  aircraft  was  required  to  depart  the  Fort  Hood 
area.  On  the  way  back,  all  components  of  the  radar  and  radar  simulation  were  rebooted  and  all 
components  functioned  normally. 

3.2.3  29  March  Trial 

The  29  March  trial  was  very  successful.  No  nodes  reported  major  problems  and  the  SAR 
simulation  (sim)  hardware  problem  at  Grumman  from  the  24  March  trial  was  resolved.  In 
addition,  the  redundancy  added  to  the  network  following  the  Phase  2  test  proved  to  be  a 
worthwhile  addition.  The  T-1  line  between  the  TCAC  and  Grumman  went  out  for  a  short  while, 
but  the  test  was  not  impacted  as  PDUs  were  routed  to  Northrop  Grumman  from  the  TCAC  via 
Fort  Hood. 

Following  the  trial  on  29  March,  it  was  determined  that  there  were  problems  with  the  Northrop 
Grumman  log  file  for  the  trials  on  25  and  29  March.  Although  the  logger  received  nearly  all  the 
PDUs,  the  log  files  showed  large  PDU  losses.  N&E  personnel  checked  the  loggers  and  log  files 
and  found  no  problems.  It  was  later  determined  that  compiling  work  accomplished  during  the 
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test  may  have  affected  the  processing  capability  of  the  computer  and  caused  the  PDU  losses.  For 
the  30  March  trial,  it  was  decided  that  no  compiling  work  would  be  done  during  the  trial. 

3.2.4  30  March  Trial 

The  trial  on  30  March  was  also  very  successful.  No  nodes  reported  any  problems,  and  there  were 
no  test  stoppages  or  delays.  The  test  was  stopped  one  hour  early  to  allow  for  analysis  of  the 
Northrop  Grumman  log  file.  The  log  file  was  complete,  and  it  was  determined  that  the  logger 
was  functioning  properly. 

3.2.5  31  March  Live  Flight 

The  live  flight  on  31  March  was  the  most  successful  of  the  three  ETE  Test  flights.  There  were 
no  communication  problems,  and  no  Janus  restarts  were  required.  SATCOM  and  SCDL  were 
both  briefly  postponed.  Following  these  delays,  the  test  ran  very  smoothly.  The  E-8C  flew  an 
extra  hour  on  station  allowing  for  extra  test  time.  Virtual  only,  live  only,  and  a  mixed  area  were 
displayed  throughout  the  on-station  period  with  the  mixed  area  containing  both  live  and  virtual 
ground  vehicles. 
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4.0  Analysis  of  Test  Objectives 

During  the  operational  test  portion  of  Phase  4  of  the  ETE  Test,  JADS  analysts  collected 
information  to  address  the  issues,  JADS  objectives,  and  test  measures  as  outlined  in  the  JADS 
Program  Test  Plan  (FTP)  and  the  ETE  Test  Data  Management  and  Analysis  Plan  (DMAP).  Only 
those  subobjectives  and  measures  evaluated  using  Phase  4  results  are  discussed. 

4.1  JADS  Issue  1.  Whatisthepresentutility  of  ADS,  including  DIS,  for  T&E? 

This  objective  determines  the  extent  to  which  ADS  technology  can  support  the  T&E  of  current 
and  future  C4ISR  systems. 

4.1.1  JADS  Objective  1-1.  Assess  the  validity  of  data  from  tests  utilizing  ADS,  including 
DIS,  during  test  execution. 

During  Phase  4,  the  ETE  Test  team  examined  the  validity  of  data  from  an  ADS  configuration 
incorporating  the  VSTARS  simulation,  Janus  simulation,  and  other  components  into  a  C4ISR 
architecture. 

JADS  Measure  l-l-O-l.  Degree  to  which  ADS  provides  valid  system  under  test  (SUT)  data. 

JADS  Measure  1-1-0-2.  Percentage  of  ADS  data  which  are  valid  (data  supporting  test 
measures  which  are  timely,  accurate,  reliable,  and  otherwise  faithfully  represent  real-world 
systems  data). 

JADS  Measure  1-1-0-3.  Degree  to  which  test  participants  were  able  to  distinguish  among 
virtual,  constructive  and  live  targets. 

JADS  Measure  1-1-0-4.  Degree  to  which  test  actions  were  impacted  because  of  the  ability 
to  distinguish  among  virtual,  constructive  and  live  targets. 

These  four  test  questions  gauge  the  ability  of  an  ADS-enhanced  live  test  environment  to  provide 
valid  data  for  a  C4ISR  system  under  test.  The  first  measure  addresses  the  validity  of  the  SUT 
output  data  that  form  the  data  elements  for  evaluating  SUT  measures.  The  second  measure 
provides  an  assessment  of  the  input  data  provided  to  the  SUT  by  the  ADS-enhanced  live  test 
environment.  The  third  and  fourth  measures  address  the  impact  on  testing  of  the  various  targets 
provided  by  an  ADS-enhanced  live  test  environment. 

These  measures  were  primarily  addressed  through  the  implementation  of  the  ETE  Test  V&V 
Plan,  and  the  Phase  3  V&V  and  system  integration  tests.  Since  JADS  is  a  DoD-sponsored  joint 
test,  the  basis  for  all  V&V  activities  was  the  DIS  Nine-Step  W&A  Process  Model  and  its 
accompanying  Recommended  Practice  for  Distributed  Interactive  Simulation  —  Verification, 
Validation,  and  Accreditation  (draft-21  May  1996).  Figure  9  provides  an  overview  of  the  W&A 
process  model.  Note  that  this  model  has  been  extensively  discussed  with  numerous  members  of 
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the  DIS  modeling  and  simulation  (M&S)  community,  has  been  generally  accepted  by  V&V 
practitioners,  and  is  currently  being  balloted  as  a  standard. 


Figure  9.  DIS  Nine-Step  VV&A  Process  Model 


The  ETE  Test  VV&A  represented  a  tailoring  and  implementation  of  the  nine-step  process  to  a 
multiservice  test  of  a  major  system.  Joint  STARS,  augmented  with  ADS.  The  tailored  ETE  Test 
process  model  used  is  described  in  the  V&V  reports  for  the  ETE  Test.  (Available  from  JADS, 
2050A  2nd  Street  SE,  Kirtland  Air  Force  Base,  New  Mexico,  87117-5522.  After  1  March  2000 
refer  requests  to  HQ  AFOTEC/HO,  8500  Gibson  Blvd.  SE,  JQrtland  Air  Force  Base,  New 
Mexico  87117-5558  or  SAIC  Technical  Library,  2001  North  Beauregard  St.  Suite  80, 
Alexandria,  Virginia  223 11.) 


The  results  from  implementing  the  ETE  Test  V&V  process  model  for  the  Phase  4  ADS 
configuration  are  detailed  in  the  ETE  Test  V&V  reports  and  are  sununarized  as  follows. 

•  Verification  of  Janus  6.88D 


-  Janus  6.88D  was  capable  of  simulating  at  least  5000  entities  (actually  9999  entities)  with 
at  least  twenty-five  percent  moving. 

-  Janus  6.88D  operators  were  capable  of  fully  interacting  with  a  scenario  while  it  was 
running.  In  particular,  the  Janus  site  representative  interacted  with  the  scenario  after 
several  ATACMS  impacts  by  altering  the  behavior  of  the  surviving  entities  within  the 
immediate  area  of  the  impact. 

-  Janus  6.88D  accepted  and  processed  scenarios  with  a  duration  longer  than  eight  hours. 

-  Janus  6.88D  was  capable  of  proceeding  at  a  pace  representative  of  near  real  time. 

-  Janus  6.88D  was  capable  of  utilizing  scenarios  conducted  upon  terrain  representing  a 
simulation  area  of  at  least  170  km  by  170  km. 
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•  Verification  of  the  RPSI 


-  The  RPSI  received  and  integrated  virtual  data  from  the  Phase  4  ADS-enhanced  live  test 
environments. 

-  The  RPSI  operated  in  three  modes:  live  only,  mixed  live  and  virtual,  and  virtual  only 
using  the  standard  Joint  STARS  MTI  message  format. 

-  The  radar  timeline  was  not  impacted  by  the  MTI  simulation. 

-  The  RPSI  processed  parameter  data  in  the  same  format  as  Joint  STARS. 

-  The  RPSI  displayed  live  SARs  in  live  areas  of  interest  and  virtual  SARs  in  virtual  areas 
using  the  standard  Joint  STARS  SAR  message  format.  Because  of  a  software  error,  the 
RPSI  did  not  display  virtual  SARs  in  a  mixed  area.  Instead,  it  displayed  live  SARs  in  the 
mixed  area.  This  problem  had  previously  existed  in  the  laboratory  version  of  the  RPSI 
that  was  integrated  into  VSTARS.  The  correction  was  simple  and  previously  tested  but 
involved  a  change  to  the  JDS  07_006+  software  build  that  the  JTF  had  already  approved 
for  flight  testing  on  15  March  1999.  Since  the  aircraft  would  not  be  available  for 
additional  testing  prior  to  the  19  March  flight  test,  and  the  feamre  would  not  be  used 
during  the  conduct  of  the  Phase  4  test,  it  was  decided  by  the  ETE  Team  lead  to  not  correct 
the  JDS  07_006+  software  build. 

-  The  RPSI  permitted  all  the  installed  operator  workstation  software  to  function  without 
abnormal  fault  messages. 

•  Verification  of  compliance  standards  (DIS  step  2).  It  was  verified  that  the  PDUs  emitted 
by  each  simulation  adhered  to  the  prescribed  format. 

-  It  was  verified  that  Janus  6.88D  issued  DIS  2.0.4  ESPDUs  that  conformed  in  content  and 
format  with  the  DIS  2.0.4  standards  as  amended  by  JADS.  (JADS  modified  the  ESPDU 
time-stamp  format  from  time  passed  since  the  beginning  of  the  current  hour  to 
milliseconds  since  the  beginning  of  the  vignette.  This  allowed  testers  to  trace  an  ESPDU 
back  to  a  discrete  event  that  occurred  within  the  Janus  vignette.) 

-  The  AFATDS  located  at  Fort  Hood  communicated  directly  with  the  AFATDS  at  Fort  Sill 
using  standard  AFATDS  message  traffic  instead  of  DIS  PDUs.  The  AFATDS  located  at 
Fort  Sill  than  communicated  through  a  DIS  NIU  with  TAFSM  using  DIS  PDUs. 

•  Verification  of  compatibility  (DIS  step  6).  It  was  verified  that  the  modeling  and  simulation 
(M&S)  components  exchanged  data  and  interacted  appropriately  with  one  another;  that 
individual  components  correctly  used  the  common  data  (e.g.,  terrain,  weather)  to  generate 
their  portion  of  the  synthetic  environment,  and  that  the  overall  implementation  was  adequate 
to  address  the  exercise  requirements.  It  was  also  verified  that  the  network  allowed  transfer  of 
information  among  the  components  without  corruption  and  the  individual  components  shared 
a  common  perspective  of  the  virtual  reality  produced  by  the  exercise. 

-  It  was  verified  that  Janus  6.88D  received  each  ESPDU,  fire  PDU,  and  detonate  PDU 
issued  to  it  by  the  network  and  took  the  appropriate  action  as  dictated  by  the  PDU.  In 
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particular,  Janus  correctly  identified  ATACMS  launches  initiated  by  Fort  Sill  that  were 
outside  the  scenario  terrain  box. 

-  It  was  verified  that  TAFSM  accepted  artillery  missions  using  DIS  PDUs  and  that  TAFSM 
issued  a  fire  and  detonate  PDU  whenever  an  ATACMS  missile  was  fired  and 
subsequently  detonated. 

•  Validation  of  the  ETE  Test  Phase  4  synthetic  environment  (SE)  (DIS  Step  7). 

-  It  was  validated  that  the  integrated  simulations  were  adequate  to  satisfy  the  ETE  Test 
requirements  such  that  the  behavior  and  performance  of  the  SE  mapped  sufficiently  to 
real-world  counterparts  for  the  specific  ETE  Test  application.  The  conclusion  that  the 
ETE  Test  SE  was  a  realistic  representation  of  the  real  world  was  based  on  structured 
interviews  with  the  RPSI,  LGSM,  and  TAG  operator  subject  matter  experts  (SME) 
participating  in  the  test. 

-  It  was  specifically  validated  that  Janus  6.88D  represented  vehicle  behavior  to  the  degree 
detectable  by  the  Joint  STARS,  as  judged  by  Joint  STARS  operator  SMEs  viewing 
vehicle  movement  presented  by  the  Joint  STARS  operator  workstation.  Realistic  vehicle 
behavior  was  achieved  after  correcting  an  anomaly  observed  during  Phase  2  ETE  risk 
reduction  testing. 

-  The  anomaly  consisted  of  portions  of  convoys  missing  turns  and  wandering  off  into 
the  desert.  The  lost  portion  of  the  convoy  would  then  jump  back  into  formation  after 
a  period  of  time  and  resume  normal  movement. 

—  It  was  determined  that  this  was  caused  by  Janus  not  sending  change  of  state  ESPDUs 
in  a  timely  fashion.  Within  Janus,  or  any  other  DIS  compliant  simulation,  ESPDUs 
are  sent  for  any  change  of  state  (starting,  stopping,  turning,  or  changing  speed  beyond 
preset  limits)  in  addition  to  the  normal  heartbeat  ESPDUs  for  stationary  entities. 
When  Janus  did  not  send  the  change  of  state  ESPDUs  in  a  timely  fashion,  VSTARS 
continued  to  move  the  vehicle  in  a  straight  line  based  upon  the  last  received  ESPDU. 

—  The  delay  was  caused  by  the  method  that  Janus  uses  to  send  ESPDUs.  Janus  cycles 
through  the  list  of  entities  at  a  rate  set  by  the  operator  and  sends  either  a  heartbeat 
ESPDU  or  a  change  of  state  ESPDU  as  appropriate.  Because  there  were  nearly 
10,000  entities  represented  within  Janus,  it  was  necessary  to  set  the  cyclic  rate  at  a 
low  enough  value  so  that  it  took  Janus  nearly  15  minutes  to  issue  the  10,000  ESPDUs. 
The  solution  was  to  turn  off  the  heartbeat  prior  to  the  beginning  of  entity  movement. 
This  allowed  adjustment  of  the  cyclic  rate  so  that  Janus  would  check  the  10,000 
entities  every  10  seconds  and  issue  only  change  of  state  ESPDUs.  This  reduced  the 
delay  in  sending  the  change  of  state  PDUs  and  the  anomaly  was  no  longer  detectable. 

-  For  the  RPSI  validation,  several  of  the  operators  who  had  participated  in  the  modified 
Turing  test  during  Phase  2  also  performed  the  SIT.  They  were  asked  to  compare  RPSI 
performance  with  the  modified  Turing  test  and  Joint  STARS.  Slight  differences  were 
noted,  but  test  participants  still  characterized  the  RPSI  as  having  the  same  capabilities  and 
limitations  as  Joint  STARS.  They  could  not  identify  any  RPSI  process  or  function  that 
limited  their  ability  to  perform  their  mission/job  or  altered  their  approach  to  their 
mission/job.  Results  demonstrated  a  close  correspondence  between  the  RPSI  and  Joint 
STARS. 
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Conclusion  for  JADS  Measure  l-l-O-l.  The  Phase  4  ADS  configuration  produced  valid  SUT 
data  and  did  not  affect  or  hamper  the  functionality  of  the  actual  Joint  STARS  radar.  The 
credibility  of  the  SUT  data  was  enhanced  by  the  use  of  actual  operational  hardware  wherever 
possible  (e.g.,  ATWS,  LGSM,  AFATDS)  with  actual  trained  operators.  The  RPSI  was  designed 
so  that  the  only  simulated  portion  was  the  simulation  of  the  MTI  and  SAR  radar  modes  within 
the  radar  subsystem;  everything  else  was  either  integration  code  or  actual  E-8C  system  code.  The 
inputs  into  the  RPSI,  except  for  the  target  data,  were  normal  inputs  into  the  real  radar  processor, 
and  the  outputs  were  the  actual  radar  reports.  The  radar  simulations  were  parallel  processes  with 
the  radar  when  live  and  virtual  were  mixed  and  solved  the  radar  equations  in  order  to  achieve  the 
required  fidelity. 

Conclusion  for  JADS  Measure  1-1-0-2.  Under  normal  operations,  all  input  data  (radar  reports) 
provided  to  the  SUT  (Joint  STARS)  by  the  ADS-dnhanced  live  test  environment  were  valid. 
Network  performance  and  reliability  in  delivering  data  to  the  SUT  are  analyzed  under  JADS 
Objective  2-1. 

Since  the  Phase  4  ADS  configuration  produced  valid  SUT  data,  this  configuration  is  usable  for 
Joint  STARS  DT&E  and  OT&E. 

Conclusion  for  JADS  Measure  1-1-0-3.  During  the  Phase  4  operational  testing  (OT),  all  GSM 
operators  and  JADS  observers  were  able  to  easily  distinguish  between  the  live  areas  and  the 
virtual  areas.  Though  the  size,  color,  contrast,  and  movement  of  the  MTI  radar  reports  were 
indistinguishable,  the  quantity  of  MTI  dots  and  the  clutter  associated  with  them  was  clearly 
distinguishable.  The  primary  reason  for  this  was  because  the  vignettes  used  by  JADS  depicted  a 
tactical  scenario,  while  the  live  areas  consisted  of  civilian  traffic  in  and  around  Fort  Hood 
(Killeen,  Texas).  The  operators  were  familiar  with  the  live  area  and  were  able  to  easily 
determine  major  roads  and  highways  around  Fort  Hood. 

If  the  live  and  virtual  areas  had  been  of  the  same  type  (tactical,  administrative),  it  would  have 
been  much  more  difficult  to  distinguish  between  the  two.  Within  the  mixed  area,  it  was 
impossible  to  distinguish  between  the  live  and  virtual  entities  based  upon  size,  color,  contrast  and 
movement.  Since  the  tactical  scenario  took  place  in  Iraq,  and  the  maps  only  showed  Iraqi  roads, 
one  could  assume  that  the  vehicles  moving  on  the  mapped  road  structure  were  virtual.  However, 
there  was  no  way  to  tell  positively  that  this  was  the  case,  and  those  vehicles  not  moving  on  the 
mapped  roads  could  easily  be  off-road  traffic  moving  about  the  desert  or  vehicles  moving  on 
unmapped  Foads. 

Conclusion  for  JADS  Measure  1 -1-0-4.  The  operators  in  the  GSM  provided  intelligence  to  the 
TAG  either  because  the  TAG  requested  it  or  because  the  operators  noticed  something  significant 
and  reported  it  to  the  TAG.  Since  the  TAG  officer  in  charge  (OIG)  was  only  concerned  about  the 
virtual  entities,  information  regarding  live  entities  was  not  requested.  Similarly,  since  the 
operators  knew  that  the  TAG  was  only  interested  in  simulated  entities,  they  did  not  request  radar 
reports  for  live  entities  on  their  own.  The  only  times  when  operators  performed  their  regular 
operator  functions  on  live  assets  were  when  the  testers  (JADS)  prompted  them.  As  a  result,  the 
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live  entities  did  not  play  any  role  in  the  GSM  to  TAG  relationship.  When  the  operators  did 
perform  operator  functions  on  live  or  simulated  assets,  their  actions  were  exactly  the  same  for 
each.  The  method  of  requesting  a  SAR,  different  types  of  MTI,  or  tracking  vehicles/convoys  was 
exactly  the  same. 

Utility  for  DT&E 

Though  not  specifically  tested  in  Phase  4,  the  ADS-enhanced  live  test  environment  can  be  used 
to  conduct  developmental  testing  of  all  nonradar  subsystems  that  comprise  Joint  STARS,  given 
that  the  RPSI  is  an  accurate  representation  of  the  radar.  Obviously,  the  RPSI  can  not  be  used  to 
conduct  developmental  testing  of  the  radar  subsystem. 

Most  developmental  testing  of  product  improvements  require  test  flights  prior  to  acceptance.  As 
an  example,  one  of  the  features  of  the  workstation  used  on  the  E-8C  is  an  automatic  tracker  (A- 
tracker).  The  A-tracker  uses  radar  reports  and,  when  initiated,  will  automatically  track  a 
designated  formation  providing  bearing,  speed  and  number  of  vehicles.  Prior  to  the  creation  of 
the  ADS-enhanced  live  test  environment  it  was  necessary  to  either  have  a  functioning  radar  (test 
flight)  or  a  recording  of  a  functioning  radar  in  order  to  test  the  A-tracker.  Test  cases  were 
basically  limited  to  those  that  could  be  achieved  at  Eglin  Air  Force  Base,  Florida,  with  a  minimal 
number  of  vehicles  traveling  under  peacetime  safety  restrictions. 

Use  of  the  ADS-enhanced  live  test  environment  allows  the  tester  to  repeat  the  live  test  described 
above,  and  at  the  same  time,  conduct  ADS-enhanced  tests  of  more  complex  scenarios  that  are 
required  but  impossible  to  achieve  at  Eglin  AFB.  The  A-tracker  could  be  tested  under 
operationally  realistic  conditions,  closing  the  gap  between  specification  testing  and  operational 
performance. 

Utility  for  OT&E 

The  utility  of  this  configuration  for  Joint  STARS  OT&E  was  evaluated  by  determining  which 
measures  from  the  Joint  STARS  Multiservice  Operational  Test  and  Evaluation  (MOT&E)  Plan^ 
could  be  supported.  Appendix  B  (available  under  separate  cover  from  JADS  JTF)  identifies 
which  Joint  STARS  MOT&E  measures  could  be  evaluated  using  the  Phase  4  ADS  configuration. 
For  comparison,  the  measures  that  were  addressed  during  the  contingency  operations'*  were  also 
identified. 


Joint  Surveillance  Target  Attack  Radar  System  (Joint  STARS)  Multiservice  Operational  Test  and  Evaluation 
(MOT&E)  Plan,  Air  Force  Operational  Test  and  Evaluation  Center,  Kirtland  Air  Force  Base,  New  Mexico,  21 
February  1995. 

Joint  Surveillance  Target  Attack  Radar  System  (Joint  STARS)  Contingency  Operations  Test  and  Evaluation 
(COT&E)  Plan,  Air  Force  Operational  Test  and  Evaluation  Center,  Kirtland  Air  Force  Base,  New  Mexico,  1 
December  1995. 
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Results  in  Appendix  B  are  summarized  as  follows. 

-  The  Phase  4  ADS  configuration  evaluated  15  out  of  45  effectiveness  measures  of 
performance  (MOP)  (including  two  MOPs  not  evaluated  during  the  Joint  STARS 
contingency  operations  test  and  evaluation)  and  all  eight  measures  of  effectiveness 
(MOE)  using  the  actual  components  of  the  system. 

-  If  additional  elements  (simulated  or  real)  were  added  to  the  Phase  4  ADS  configuration  it 
could  be  used  to  fully  evaluate  18  of  the  18  MOPs  supporting  critical  operational  issue 
(COI)-l.  {Does  Joint  STARS  perform  its  tactical  battlefield  surveillance  mission?) 

-  If  additional  elements  (simulated  or  real)  were  added  to  the  Phase  4  ADS  configuration  it 
could  be  used  to  fully  evaluate  24  of  the  24  MOPs  supporting  COl-2.  {Does  Joint  STARS 
support  the  execution  of  attacks  against  detected  targets!) 

-  The  Phase  4  ADS  configuration  can  support  evaluation  of  the  single  measure  of 
effectiveness  (MOE)  for  COI-3  {Does  Joint  STARS  provide  timely  and  accurate 
information  to  support  battlefield  management  and  target  selection!)  and  both  MOPs  for 
001-4.  {Can  the  Joint  STARS  system  be  sustained  in  an  operational  environment?) 

-  If  additional  elements  (simulated  or  real)  were  added  to  the  Phase  4  ADS  configuration  it 
could  be  used  to  support  the  evaluation  of  17  of  the  17  additional  Joint  STARS 
effectiveness  measures. 

-  Since  the  Phase  4  ADS  configuration  utilizes  an  actual  GSM,  27  out  of  27  suitability 
MOPs  involving  the  GSM  and  the  E-8C  could  be  evaluated. 

-  The  Phase  4  ADS  configuration  could  support  the  evaluation  of  all  8  human  factors 
measures. 

-  The  Phase  4  ADS  configuration  could  not  support  the  evaluation  of  any  of  the  E-8C 
software  evaluation  measures  (0  of  6). 

In  summary,  the  Phase  4  ADS  configuration  evaluated  the  same  measures  as  the  Phase  2  ADS 
configuration  using  the  actual  components  of  the  system.  However,  if  additional  elements 
(simulated  or  real)  were  added  to  the  Phase  4  ADS  configuration,  it  could  evaluate  all  45 
effectiveness  MOPs  (including  two  MOPs  not  evaluated  during  the  contingency  operations)  and 
all  8  effectiveness  MOEs.  Furthermore,  the  augmented  Phase  4  ADS  configuration  could  be 
used  to  evaluate  the  GSM  and  E-8C  suitability  MOPs  (27  out  of  27  suitability  MOPs),  all  of  the 
human  factors  MOPs,  and  all  of  the  software  MOPs. 

In  addition,  for  those  MOPs  and  MOEs  that  address  Joint  STARS  effectiveness  in  an  operational 
environment,  the  Phase  4  ADS  configuration  provided  a  more  complete  and  realistic  targeting 
environment  and  the  ability  to  engage  targets  as  desired. 
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4.1.2  JADS  Objective  1-2.  Assess  the  benefits  of  using  ADS,  including  DIS,  in  T&E. 

4. 1.2.1  JADS  Subobjective  1-2-1.  Assess  ADS  capability  to  support  the  early  phases  of  the 
acquisition  process. 


JADS  Measure  1-2-1 -4.  Degree  to  which  ADS  can  improve  early  operational  assessments. 

One  of  the  common  shortcomings  of  operational  testing  for  C4I  systems  is  the  inability  to 
conduct  a  robust  initial  operational  test  and  evaluation  (lOT&E)  of  the  complete  systems  of 
systems.  This  shortcoming  all  too  often  results  in  the  execution  of  a  final  operational  evaluation 
(OPEVAL)  that  uncovers  significant  problems  with  the  system  under  test.  These  problems  can 
limit  the  tester  in  evaluating  the  system’s  operational  effectiveness  and  suitability.  In  severe 
cases  major  portions  or  even  the  entire  test  event  must  be  rescheduled  and  repeated  after 
deficiencies  have  been  corrected. 

ADS  can  provide  a  useful  tool  for  the  tester  in  support  of  early  operational  testing.  ADS  can 
bring  together  the  components  of  a  complex  C4I  system  of  systems  for  testing  using  a  mix  of 
actual  and  simulated  assets.  This  capability  allows  for  earlier  testing  which  may  identify 
potential  issues  that  would  impact  the  conduct  or  the  results  of  an  OPEVAL. 

4. 1.2.2  JADS  Subobjective  1-2-2.  Assess  ADS  capability  to  support  T&E  planning  and  test 
rehearsal. 

JADS  Measure  1-2-2-1.  Degree  to  which  a  test  concept/design  can  be  improved  through 
experimentation  using  an  ADS  environment. 

JADS  Measure  1 -2-2-2  Degree  to  which  a  test  concept/design  is  impacted  by  ADS. 

ADS  has  the  ability  to  greatly  impact  test  concept  and  design  allowing  the  test  designer  to 
surmount  such  conventional  testing  shortfalls  as  inadequate  numbers  of  friendly  and  threat 
systems,  safety,  range  availability,  and  environmental  conditions.  Simulations  can  adequately 
depict  these  conditions  and  allow  the  testers  to  adequately  test  concepts  that  they  might  not  be 
able  to  otherwise  test.  Since  an  ADS-enhanced  live  test  environment  does  not  require  massive 
coordination  of  soldiers  and  movement,  it  is  much  easier  to  make  changes  to  the  test  concept  and 
record  the  results.  The  ETE  Test  configuration  was  established  specifically  for  the  ETE  Test 
concept.  Similarly,  other  test  configurations  could  be  established  to  expand  on  the  ETE  Test 
concept  or  to  test  new  ones. 

There  are  some  negative  impacts  on  test  concept  and  design  using  an  ADS-enhanced  live  test 
environment.  First,  technical  expertise  is  required  to  establish,  maintain,  and  troubleshoot  ADS- 
related  problems.  These  include  personnel  trained  to  operate  the  different  simulations  as  well  as 
to  maintain  the  network  linking  different  sites.  Second,  establishing  an  ADS  configuration  has  a 
substantial  up-front  cost.  Finally,  meticulous  coordination  is  required  to  schedule  and  execute  an 
ADS  test,  since  problems  at  one  node  will  almost  certainly  affect  the  execution  of  the  test 
concept  at  other  nodes.  For  example,  VSTARS  malfunctions  on  several  days  greatly  diminished 
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our  ability  to  test  and  record  actions  at  the  TAG,  consequently  affecting  the  TAG’S  ability  to 
identify  targets  and  use  ATAGMS  properly  to  destroy  those  targets. 

Testers  must  evaluate  the  risks  and  rewards  of  the  test  concept  and  design  before  testing.  Though 
ADS  allows  for  a  more  robust  test  design,  the  cost,  technical  expertise,  and  coordination  required 
may  make  it  very  difficult  or  impractical. 

JADS  Measure  1-2-2-4.  Degree  to  which  pretest  exercise  of  data  reduction  and  analysis 
routines  using  ADS  improved  test  preparations. 

JADS  Measure  1-2-2-7.  Degree  to  which  data  reduction  and  analysis  routines  can  be 
improved  through  experimentation  using  an  ADS  environment. 

JADS  Measure  1-2-2-8.  Degree  to  which  data  reduction  and  analysis  routines  are  impacted 
by  incorporating  ADS  as  part  of  the  test  concept/design. 

These  measures  evaluated  the  impact  of  ADS  technology  on  data  reduction  and  analysis  routines. 
In  support  of  these  test  questions,  the  ETE  Test  analysts  conducted  interviews  with  test  team 
members  involved  in  data  reduction  and  analysis  during  the  Phase  4  operational  test  and  test 
rehearsals. 

The  Phase  4  data  reduction  and  analysis  methodology  were  based  in  large  part  on  procedures 
successfully  used  in  earlier  JADS  tests,  with  the  exception  of  the  reduction  and  analysis  of  the 
SATGOM  data.  Procedures  used  during  the  functionality  and  integration  (FI)  and  risk  reduction 
tests,  as  well  as  during  the  Phase  4  operational  test  (OT),  were  basically  unchanged  from  those 
used  previously.  During  Phase  3,  pretest  exercises  and  experimentation  were  accomplished  on 
the  data  reduction  and  analysis  required  for  the  SATGOM  data  in  order  to  determine  the 
necessary  changes  to  the  test  configuration.  Based  on  these  pretest  activities,  the  Janus  scenarios 
were  adjusted  to  ensure  that  the  link  capacity  was  not  exceeded.  The  incorporation  of  the  E-8G 
into  the  ETE  Test  environment  also  impacted  the  data  reduction  and  analysis  methodology.  In 
particular,  the  conversion  of  the  data  from  the  ground  NIU  at  Northrop  Grumman  and  the  data 
from  the  logging  software  on  the  E-8G  into  a  usable  format  required  a  great  deal  of  manipulation 
by  the  JADS  analyst  programmers. 

Data  reduction  and  analysis  activities  in  Phase  4  were  aided  by  the  use  of  the  JADS  Analysis 
Toolbox.  The  routines  in  the  toolbox  were  also  useful  in  analyzing  the  SATGOM  link  data  once 
the  data  were  manipulated  and  converted  into  a  format  similar  to  that  of  the  PDUs. 

Developed  over  the  course  of  earlier  JADS  tests,  the  toolbox  is  a  set  of  software  routines  written 
in  the  G++  programming  language  and  integrated  into  a  single  user  interface.  The  toolbox  can  be 
employed  to  develop  tables  and  plots  of  PDU  data  in  near  real  time.  After  the  test,  toolbox  users 
can  replay  or  get  selected  data  in  a  text-readable  format  from  JADS  log  files  and  obtain  plots  and 
tables  of  PDU  statistics.  The  PDU  statistics,  in  turn,  include  the  number  of  PDUs  received,  the 
number  of  each  type  of  PDU  received,  and  the  rate  at  which  PDUs  are  being  received.  An 
important  point  is  that  the  ETE  Test  log  files  were  larger  than  earlier  JADS  test  log  files. 
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Modifications  were  made  to  the  toolbox  software  to  minimize  the  impact  of  the  size  of  the  log 
files  on  file  access  time  as  well  as  on  the  amount  of  memory  needed  for  calculations.  (For 
further  information  on  the  JADS  Analysis  Toolbox,  contact  the  JADS  JTF  or  the  JADS  Web  site 
at  http;//www .jads.abq.com  where  it  will  be  available  until  March  2001.  After  that  it  will  be 
available  at  HQ  AFOTEC/HO,  8500  Gibson  Blvd.  SE,  Kirtland  Air  Force  Base,  New  Mexico 
87117-5558  or  SAIC  Technical  Library,  2001  North  Beauregard  St.  Suite  8(X),  Alexandria, 
Virginia  223 11.) 

Table  2  describes  the  estimated  number  of  hours  spent  on  rehearsing  data  reduction  and  analysis 
procedures  for  the  Phase  4  risk  reduction  tests  and  OT.  The  times  presented  in  Table  2  do  not 
include  time  spent  on  either  data  collection  or  report  writing. 

Table  2.  Time  Spent  on  Pretest  Data  Reduction  and  Analysis  Rehearsals 


Data  Type 

Risk  Reduction 
Testing 

Operational 

Testing 

PDUData 

12hrs 

16  hrs 

Latency 

12hrs 

16  hrs 

Network 

Availability 

6  hrs 

8  hrs 

ADS  System 
Availability 

6  hrs 

8  hrs 

Bandwidth 

3  hrs 

4  hrs 

Packet  Rate 

2  hrs 

3  hrs 

Other  OT 

Test  Measures 

20  hrs 

25  hrs 

The  ETE  Test  Phase  4  also  demonstrated  the  unique  advantages  of  an  ADS-enhanced  live  test 
environment  with  respect  to  data  reduction  and  analysis  processes. 

—  For  both  ADS  and  conventional  tests,  these  routines  are  typically  developed  prior  to  the 
actual  test  events.  For  non-ADS  tests,  actual  test  assets  (i.e.,  flight  time)  may  need  to  be 
expended  in  order  to  produce  the  data  needed  to  test  these  routines.  In  contrast,  the  JADS 
analysts  were  able  to  exercise  their  data  reduction  and  analysis  routines  using  data  from  the 
FI  and  risk  reduction  tests  accomplished  prior  to  Phase  4  OT  testing.  This  early  availability 
of  test  data  enabled  the  analysts  to  check  out  and  refine  these  procedures  and  prevented  the 
loss  of  critical  test  data  from  the  actual  Phase  4  OT  test. 

—  If  an  ADS  test  team  emphasizes  the  early  exercise  of  data  reduction  and  analysis  routines 
during  pretest  rehearsals,  it  can  reduce  or  eliminate  the  loss  of  critical  data  and  expenditure  of 
valuable  test  assets  during  subsequent  actual  testing.  Even  the  addition  of  a  new  type  of  data 
(i.e.,  SATCOM  link  data)  can  be  smoothly  managed  by  relying  on  the  data  reduction  analysis 
methodology  and  routines  that  have  been  successful  in  the  past. 


38 


-  The  networked  nature  of  an  ADS-enhanced  live  test  environment  provides  a  built-in  means  to 
support  data  management.  During  the  risk  reduction  tests  prior  to  Phase  4  operational 
testing,  the  ETE  Test  team  took  advantage  of  this  capability  and  was  able  to  speed  the 
transport  of  test  data  from  the  nodes  to  a  central  collection  and  analysis  point. 

JADS  Measure  1-2-2-5.  Degree  to  which  ADS  can  be  used  for  tactics  development  prior  to 
test  execution. 

JADS  Measure  1-2-2-9.  Degree  to  which  tactics  development  can  be  improved  by 
experimentation  in  an  ADS  environment. 

These  measures  determined  if  the  techniques  and  procedures  used  by  the  test  participants  to 
obtain  and  disseminate  information  on  target  enemy  forces  could  be  enhanced  in  any  way  using 
an  ADS-enhanced  live  test  environment.  In  a  broader  context,  JADS  analysts  wanted  to 
determine  if  the  ADS  test  environment  could  be  used  to  correct,  refine,  and  change  tactics, 
techniques,  and/or  procedures  prior  to  exercising  a  live  test.  During  the  previous  tests  (Phase  2 
rehearsal.  Phase  2  OT,  and  Phase  4  rehearsal),  the  TAG  QIC  was  able  to  experiment  with 
employing  the  ATACMS  based  on  intelligence  from  different  sources  (GSM  MTI,  GSM  SAR, 
Joint  Exercise  Support  System  Intelligence  Module  [JIM]  feeds)  against  different  types  of  targets 
(convoys,  assembly  areas,  logistical  sites,  etc.).  With  increased  test  time  and  experimentation, 
the  TAG  OIG  was  able  to  refine  tactics  and  confirm  its  effectiveness  during  the  Phase  4  OT. 
This  experimentation  clearly  improved  the  ability  to  select  the  right  types  of  targets  based  on  the 
right  source  of  intelligence  for  those  targets  in  order  to  maximize  the  effectiveness  of  the 
ATAGMS. 

The  increased  test  time  provided  by  an  ADS-enhanced  live  test  environment  appears  valuable  in 
allowing  G4ISR  system  operators  and  end  users  to  confirm  and  refine  current  tactics  and  to 
experiment  with  “what  if’  scenarios  and  new  tactics. 

JADS  Measure  1-2-2-6.  Determine  the  impact  of  ADS  on  test  planning  and  rehearsal  costs. 

The  use  of  ADS  in  test  execution  and  test  rehearsal  events  can  impact  overall  test  program  cost. 
This  impact  on  cost,  as  with  any  test  program,  must  be  considered  along  with  the  benefits.  The 
possible  increase  in  cost  to  a  program  from  the  use  of  ADS  can  be  justified  if  acceptable 
reductions  in  risk  or  schedule  can  be  achieved.  Risk  reduction  can  be  achieved  using  ADS  by 
allowing  a  tester  to  test  and  exercise  a  complex  system  of  systems  early  in  that  program’s  life. 
ADS  can  provide  a  tester  with  the  chance  to  rehearse  a  complex  test,  with  all  players  in  a 
distributed  manner  prior  to  bringing  actual  assets  together  for  an  expensive  test  event.  With 
many  conventional  tests,  the  complex  systems  are  not  tested  together  until  lOT&E.  If  problems 
are  encountered  in  this  late  stage,  it  is  difficult  and  very  expensive  to  make  changes.  The  earlier 
problems  are  found,  the  less  risk  to  both  program  cost  and  schedule.  ADS,  in  most  cases,  will 
not  reduce  the  cost  of  testing,  but  it  is  a  means  for  providing  a  more  robust  test  event  that  can 
minimize  cost  and  schedule  overruns. 
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4.1. 2.3  JADS  Subobjective  1-2-3.  Assess  ADS  capability  to  support  T&E  execution. 

JADS  Measure  1-2-3-1.  Degree  to  which  ADS  can  add  test  articles  to  test  execution. 

JADS  Measure  1-2-3-8.  Degree  to  which  ADS  can  provide  for  more  representative  force 
levels. 

JADS  Measure  1-2-3-17.  Degree  to  which  ADS  can  increase  the  number  of  simulation 
entities. 

JADS  Measure  1-2-3-28.  Degree  to  which  ADS  overcomes  the  use  of  improper  threats  used 
in  traditional  T&E. 

Conventional  T&E  typically  suffers  from  an  inadequate  number  of  entities.  Typical  tests  may 
have  to  be  conducted  in  conjunction  with  training  missions  in  order  to  acquire  possibly  hundreds 
of  assets.  However,  testers  will  then  have  only  minimal  control  over  test  specifics  and  must  base 
their  test  on  the  training  scenario  being  executed.  In  many  cases,  the  training  scenario  will  not 
include  the  desired  threats,  leading  to  the  improper  use  of  threats  for  the  test. 

In  contrast.  Table  3  shows  that  the  Janus  simulation  provided  thousands  of  entities  for  each 
vignette  in  Phase  4;  entities  created  specifically  to  represent  realistic  threats  for  the  test. 

Phase  4  results  imply  the  following  benefits  of  ADS-based  testing. 

—  An  ADS -enhanced  live  test  environment  using  validated  simulations  can  provide  more 
realistic  threats  and  force  levels  than  those  offered  by  conventional  tests,  i.e.,  threatsAevels 
otherwise  unavailable  because  of  cost  restrictions,  unobtainable  threats,  etc. 

—  C4ISR  system  testers  can  tailor  the  simulation  entities  operating  in  the  ADS  environment  to 
closely  reflect  the  forces  expected  in  operational  theaters,  thus  further  increasing  the 
relevance  of  the  collected  test  data. 

—  ADS  allows  testers  to  have  more  control  over  the  specific  aspects  of  the  scenarios  of  interest 
and  to  expand  their  test  concept  and  design.  A  typical  constraint  to  test  concept  development 
is  the  number  and  t)q)es  of  units  readily  available  for  a  test.  For  example,  a  conventional  test 
may  require  the  use  of  a  battalion,  but  because  of  a  prior  commitment  or  cost  these  personnel 
may  not  be  available.  In  contrast,  ADS  allows  testers  to  create  a  virtual  battalion  and  to  test 
their  concepts  with  a  minimum  number  of  personnel  and  equipment. 
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Table  3.  Entity  Count  Per  Vignette 


Vignette 

Number  of 
Entities 

2 

9,757 

3 

9,904 

4 

9,781 

5 

9,950 

JADS  Measure  1-2-3-3.  Degree  to  which  ADS  overcomes  performance  restrictions. 

ADS  is  helpful  in  simulating  maneuver  scenarios  that  overcome  shortfalls  associated  with 
traditional  testing.  This  technology  increases  test  robustness  by  providing  the  capability  for 
adding  large  numbers  of  assets.  ADS  simulations  can  depict  such  unsafe  conditions  as  convoy 
vehicles  driving  across  rugged  or  restricted  terrain  under  wartime  conditions.  Finally,  this 
technology  frees  the  tester  from  the  constraints  of  environmental  restrictions.  Without  the 
benefits  of  ADS-enhanced  testing,  it  would  be  almost  impossible  to  instrument  and  maneuver  a 
corps-size  element. 

JADS  Measure  1-2-3-4.  Degree  to  which  ADS  overcomes  the  traditional  T&E  shortfall  of 
iiisufRcient  battlespace. 

ADS  technology  can  eliminate  the  conventional  testing  disadvantage  of  insufficient  battlespace. 
The  largest  tests  conducted  by  the  Army,  comparable  to  the  scenarios  tested  by  the  ETE  Test 
team,  are  accomplished  at  the  National  Training  Center  (NTC).  These  tests  are  executed  in 
conjunction  with  NTC  using  a  maximum  battlespace  of  about  100  square  kilometers  (km^).  The 
results  show  the  great  increase  in  battlespace  exhibited  by  the  Phase  4  test  over  traditional  testing 
where  an  additional  50  km^  were  added  to  the  battlespace.  In  addition,  the  battlespace  of  150 
km^  used  by  the  ETE  Test  team  is  far  from  the  maximum  that  ADS  is  capable  of  supporting. 
This  battlespace  was  picked  for  its  applicability  to  the  scenarios  used  during  the  Phase  4  test  and 
could  be  greatly  expanded  if  necessary. 

JADS  Measure  1-2-3-7.  Degree  to  which  ADS  overcomes  the  lack  of  systems  for 
interoperability  testing  associated  with  traditional  T&E. 

JADS  Measure  1-2-3-12.  Degree  to  which  ADS  can  increase  the  number  of  systems  for 
compatibility  evaluation. 

There  were  many  interoperability  problems  among  the  fielded  real  systems  used  in  the  ETE  Test. 
Up  to  the  time  that  the  ETE  Test  occurred,  the  TAC  had  never  had  the  opportunity  to  accomplish 
interoperability  testing  under  tactical  conditions  in  a  doctrinally  correct  manner.  During  the  ETE 
Test,  several  systems  were  able  to  operate  together  and  function  as  intended  during  combat-like 
conditions.  Specifically,  the  LGSM  was  electronically  linked  to  the  ASAS  workstations  located 
within  the  TAC  and  provided  a  complete  operational  picture  of  an  enemy  corps  rear  area.  In 
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addition,  the  ASAS  workstations  were  electronically  linked  to  the  AFATDS  terminal  and 
electronically  passed  fire  missions  for  prosecution  by  the  ATACMS.  Observer  SMEs  from  El 
Corps  intelligence  community  stated  that  this  was  the  first  time  they  had  seen  all  these  systems 
linked  and  operating  as  they  should  during  a  conflict. 

JADS  Measure  1-2-3-9.  Degree  to  which  ADS  increases  the  number  of  test  events. 

Using  ADS,  testers  can  replicate  a  test,  as  well  as  test  for  longer  periods  of  time.  During  the 
Phase  4  test,  the  ETE  Test  team  was  able  to  test  for  50  hours  over  a  period  of  eight  test  days.  A 
test  using  real  assets  of  similar  duration  would  be  very  expensive  and  probably  even  impossible. 

JADS  Measure  1-2-3-16.  Degree  to  which  ADS  can  overcome  test  area  time/availability 
restrictions. 

ADS  technology  can  overcome  the  shortfalls  of  limited  test  area  time  and  availability  often 
associated  with  traditional  testing.  Conventional  testing  requires  the  use  of  live  ranges,  soldiers 
and  equipment  to  depict  realistic  conditions  for  testing  and  training.  Typically,  these  resources 
have  to  be  scheduled  far  in  advance  and  are  available  only  for  a  limited  time  or  usage.  An  ADS- 
enhanced  test  can  surmount  this  shortcoming  by  using  and  linking  simulations  as  a  substitute  for 
the  necessary  resources.  For  example,  the  ETE  Phase  4  OT  used  vignettes  to  depict  a  simulated 
Iraqi  corps  area  of  operations  that  did  not  require  a  training  range,  troops  or  equipment. 
Similarly,  other  training  and  testing  could  be  conducted  using  simulations  without  the 
coordination  and  use  of  r2inges  or  soldiers. 

JADS  Measure  1-2-3-18.  Degree  to  which  ADS  can  provide  for  improved  modeling  and 
simulation  (M&S)  used  for  testing. 

JADS  Measure  1-2-3-23.  Degree  to  which  ADS  can  provide  high  fidelity  emulators  and 
simulators  used  as  targets  and/or  threats. 

ADS  technology  can  provide  vastly  improved  models  and  simulations  for  use  in  testing.  Because 
of  the  distributed  nature  of  ADS,  the  tester  no  longer  has  to  have  all  test  assets  physically  located 
in  a  single  location.  This  saves  the  expense  of  purchasing  the  models  and  simulations  and  the 
difficulty  involved  in  manning  the  simulated  systems.  Using  the  ETE  Test  as  an  example,  if  the 
Army  desired  to  test  any  Army  components  used  in  the  test,  they  could  obtain  radar  reports  from 
VSTARS  without  owning  and  manning  (with  Air  Force  personnel)  their  own  VSTARS.  ADS 
technology  literally  makes  models  and  simulations  a  phone  call  away. 

ADS  also  allows  the  tester  to  assemble,  using  models,  simulations,  emulators,  and  fielded 
systems,  systems  of  systems  that  either  have  not  yet  been  built  or  would  not  exist  except  in  time 
of  war.  Again  using  the  ETE  Test  as  an  example,  the  first  time  that  an  ATACMS  missile  was 
fired  at  an  enemy  force,  based  on  operationally  realistic  intelligence  collected  by  Joint  STARS 
and  processed  by  an  element  of  ASAS,  was  during  the  ETE  Test.  Yet  this  is  a  fielded  system  of 
systems.  Added  to  this  is  the  battle  damage  assessment  provided  by  Joint  STARS  observing  the 
hulks  left  behind  and  the  fleeing  survivors  and  the  fact  that  all  components  were  electronically 
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linked  and  functioning  as  they  would  in  combat.  Using  this  ADS  environment,  it  is  a  simple  task 
to  add  the  next  generation  ATACMS  or  ASAS  and  find  out  how  it  will  function  as  a  component 
system  in  this  system  of  systems. 

C4ISR  system  testing  can  involve  many  complex  and  expensive  assets  that  must  be  brought 
together  for  testing.  Frequently,  the  first  time  that  all  the  systems  are  together  may  be  during  OT, 
which  means  that  problems  may  not  be  found  and  corrected  until  that  stage  of  testing.  Using 
ADS,  a  tester  can  bring  together  the  actual  assets  or  model/simulation  assets  before  OT,  and  thus 
fully  exercise  a  system  and  identify  deficiencies. 

By  providing  inputs  via  model/simulation  assets,  ADS  enables  the  tester  to  satisfy  a  test 
requirement  limited  by  cost  or  safety  concerns. 

ADS  technology  is  flexible  enough  to  support  a  wide  range  of  simulation  fidelity  requirements 
depending  on  the  type  of  testing  being  accomplished.  (For  example,  if  a  system  under  test  is 
conducting  early  developmental  testing,  the  fidelity  requirements  may  be  significantly  lower  than 
a  system  approaching  OT.) 

Lastly,  ADS  can  easily  provide  models  and  simulations  that  represent  the  threats  of  today  and  the 
threats  of  tomorrow.  All  models  and  simulations  of  threats,  no  matter  what  their  fidelity,  are  data 
driven.  The  fidelity  is  determined  by  the  accuracy  and  detail  contained  within  the  data  and  the 
fidelity  of  the  algorithms  that  use  the  data  to  depict  the  threat.  During  the  ETE  Test,  the  threats 
used  were  those  currently  fielded  within  the  Iraqi  Army.  They  could  just  as  easily  have  been  low 
observable  threats  from  a  future  battlefield.  Just  change  the  data  (radar  cross  section)  and  the 
radar  reports  change  drastically  to  reflect  a  potential  future  battlefield. 

JADS  Measure  1-2-3-19.  Degree  to  which  ADS  can  improve  upon  constrained  concept  of 
operations. 

The  concept  of  operations  in  a  conventional  test  can  be  limited  by  such  factors  as  size,  threat, 
topography,  and  environmental  considerations.  Thus,  certain  MOEs  and  MOPS  may  not  be 
evaluated  because  of  hazards  to  soldiers  and  equipment,  insufficient  numbers  of  soldiers  and 
equipment,  or  weather  and  terrain  considerations.  Using  simulations  in  an  ADS  environment  can 
overcome  many  of  these  difficulties.  Simulations  can  be  created  to  depict  how  the  SUT  may 
perform  in  a  robust  environment  with  sufficient  assets  and  under  unsafe  conditions  without  the 
restrictions  of  weather  and  terrain.  For  example,  one  scenario  created  for  A-tracker  testing 
consisted  of  hundreds  of  vehicles  in  a  tight  line  abreast  formation  traveling  at  greater  than  normal 
military  speeds  and  conducting  difficult  maneuvers  over  a  very  large  area.  For  various  reasons, 
the  specifications  that  drove  this  scenario  would  never  be  able  to  be  tested  in  a  live  test. 
Distributed  testing  also  allows  for  the  use  of  nodes  at  different  sites  to  test  the  same  SUT. 
Soldiers  and  equipment  don’t  have  to  be  deployed  to  a  common  area,  which  reduces  the 
coordination,  logistics,  and  cost  of  testing  and  training.  The  concept  of  operations  can  be  easily 
expanded  to  fully  assess  the  system  under  test. 
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JADS  Measure  1-2-3-24.  Degree  to  which  ADS  can  enable  the  use  of  correct  techniques 
and  procedures. 

Prior  to  the  use  of  ADS  in  the  ETE  Test,  intelligence  data  from  Joint  STARS,  if  any,  were 
usually  supplied  to  the  AS  AS  in  the  form  of  verbal  or  written  reports.  This  was  because  without 
ADS  there  was  no  way  to  link  Joint  STARS  to  the  test  scenario  and  provide  operationally 
significant  radar  reports.  This  was  an  incorrect  technique  and/or  procedure.  The  ground-based 
portion  of  Joint  STARS,  the  LGSM,  was  supposed  to  be  electronically  linked  to  the  ASAS 
workstation  and  radar  reports  passed  to  the  analysts  working  within  the  analysis  and  control 
element  (ACE).  The  use  of  ADS  in  the  ETE  Test  enabled  the  use  of  correct  techniques  and 
procedures. 

JADS  Measure  1-2-3-25.  Degree  to  which  ADS  can  ensure  realistic  test  scenarios. 

Conventional  T&E  typically  suffers  from  inadequate  test  scenarios  because  of  the  limited  number 
of  entities  and  the  available  exercise  area.  In  contrast,  neither  the  number  of  entities  nor  their 
movement  limit  simulations  used  in  an  ADS  environment.  For  example,  the  Phase  4  scenarios 
depicted  the  key  elements  of  several  Iraqi  corps  conducting  offensive  and  defensive  operations. 
They  included  almost  10,000  entities  and  the  entire  array  of  personnel  and  equipment  assigned  to 
a  corps-size  element.  The  layout  and  maneuver  of  the  Iraqi  corps  were  created  jusing  doctrinal 
templates  and  portrayed  tactical  movements.  According  to  the  LGSM  soldiers,  the  scenarios  in 
the  ETE  test  were  the  most  realistic  that  they  had  ever  used  for  training  or  testing. 

As  in  the  ETE  Test,  realistic  scenarios  can  be  created  using  simulations  for  many  different  types 
of  tests.  The  technology  required  to  make  them  realistic,  and  the  software  and  hardware  to 
distribute  them  to  different  sites  already  exist. 

JADS  Measure  1-2-3-26.  Degree  to  which  ADS  can  facilitate  model/simulation  data 
collection. 

Although  ADS  does  not  impact  the  internal  data  collection  of  a  model/simulation  node,  it  can 
impact  the  accessibility  of  critical  data.  ADS  technology  can  enable  the  transfer  of  large  amounts 
of  data  from  one  node  to  another  in  a  timely  and  secure  manner  using  the  same  transmission  lines 
employed  in  the  test  architecture.  This  capability  for  distributed  access  to  data  can  reduce  the 
need  to  manipulate  or  analyze  data  at  each  individual  network  node.  In  addition,  ADS  tools 
allow  for  the  monitoring  of  data  going  to  and  from  a  site  during  a  test  event,  either  in  real  or  near 
real  time.  Thus,  the  tester  can  observe  and  analyze  key  data  elements  during  a  test,  making 
changes  to  the  test  if  necessary,  hi  contrast,  the  typical  conventional  test  must  be  completed 
before  all  data  can  be  accessed  and  problems  identified. 

JADS  Measure  1-2-3-29.  Degree  to  which  ADS  can  increase  test  resources  (supplies). 

The  test  resources  (supplies)  involved  in  the  setup  and  execution  of  the  Phase  4  test  were 
examined  to  determine  which  resources  would  not  have  been  required  for  a  traditional  test.  In 
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addition,  the  amount  of  resources  (supplies)  required  for  the  test,  due  solely  to  ADS-specific 
requirements,  was  documented. 

ADS  tests,  unlike  traditional  tests,  require  resources  (supplies)  related  to  the  network  and 
distributed  nodes.  A  great  deal  of  additional  network  and  communication  equipment  is  needed  to 
ensure  network  reliability  and  good  communication  at  all  nodes.  In  addition,  an  increased 
amount  of  computer  equipment  is  required  at  the  distributed  nodes  to  run  the  simulations 
involved  in  ADS  tests.  Any  support  materials  required  at  the  distributed  nodes  would  also 
account  for  an  increase  in  test  resources  (supplies)  for  ADS  tests. 

JADS  Measure  1-2-3-30.  Degree  to  which  ADS  can  represent  indirect  fire  tactics  and 
capabilities. 

ADS  can  realistically  portray  indirect  fire  tactics  and  capabilities.  The  ETE  Test  Phase  4 
configuration  used  the  AFATDS  command  and  control  system,  collocated  with  the  TAC  at  Fort 
Hood,  to  request  fire  missions.  TAFSM,  located  at  Fort  Sill,  processed  the  fire  missions  and 
executed  them.  When  the  ATACMS  missile  was  fired  in  TAFSM,  a  fire  PDU  was  broadcast 
from  TAFSM.  The  Janus  operator,  located  at  TRAC-WSMR,  was  alerted  that  a  mission  was 
underway  so  that,  if  desired,  the  operator  could  zoom  in  on  the  strike  area  and  observe  the 
effects.  At  the  appropriate  time,  a  detonate  PDU  was  issued  by  TAFSM  giving  the  calculated 
spill  location  for  the  missile’s  bomblets.  Janus  than  calculated  the  spill  pattern  for  the  bomblets 
and  assessed  the  appropriate  damage  to  the  targets  located  within  the  spill’s  footprint.  Once  the 
damage  was  assessed,  Janus  than  broadcast  ESPDUs  reflecting  the  effects  of  the  missile  strike. 
This  was  a  doctrinally  correct  representation  of  an  ATACMS'  missile  strike  using  approved 
missile  flyout  algorithms  and  approved  weapons  effects  algorithms. 

JADS  Measure  1-2-3-31.  Degree  to  which  ADS  can  represent  joint/combined  operations 
and  capabilities. 

ADS  can  realistically  portray  joint/combined  operations.  The  Phase  4  ETE  OT  was  a  joint 
operation  that  employed  a  mix  of  Air  Force  and  Army  personnel  located  at  different  facilities 
successfully  simulating  a  C4ISR  system  interacting  with  ground-based  units.  Given  the 
necessary  planning  and  resources,  ADS  could  also  represent  combined  operations  between  forces 
of  two  or  more  allies.  The  simulation  used,  Janus,  is  capable  of  portraying  up  to  six  different 
categories  of  forces.  Since  the  scenario  was  set  in  Iraq,  it  would  have  been  a  simple  matter  to 
include  forces  from  Kuwait  in  the  scenario  thereby  creating  a  combined  operation. 

JADS  Measure  1-2-3-34.  Degree  to  which  ADS  can  provide  for  real-time  analysis. 

This  measure  examines  the  ability  of  ADS  test  participants  to  conduct  real-time  analysis  during 
testing.  For  the  Phase  4  test,  network  monitoring  tools  and  the  Janus/logger  data  collection 
capabilities  provided  data  to  JADS  analysts  allowing  for  limited  real-time  analysis.  These  tools 
provided  data  on  the  network  links,  PDU  rates,  types  of  PDUs  being  passed,  and  the  total  number 
of  PDUs  logged  at  each  node.  Analysis  of  these  data  during  Phase  4  trials  was  important  in 
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ensuring  that  the  test  was  functioning  properly  with  all  nodes  passing  the  expected  amount  and 
type  of  data. 

This  real-time  analysis  was  limited  by  the  analysts’  inability  to  manipulate  the  log  files  during 
testing,  using  tools  in  the  JADS  Analysis  Toolbox.  The  log  files  could  not  be  manipulated 
during  testing  without  losing  test  data.  In  addition,  the  initial  analysis  of  these  log  files  had  to  be 
delayed  until  immediately  after  each  Phase  4  trial. 

JADS  Measure  1-2-3-36.  Degree  to  which  ADS  can  increase  personnel  resources. 

The  duties  of  all  personnel  involved  in  the  setup  and  execution  of  the  Phase  4  test  were  examined 
to  determine  which  personnel  would  not  have  been  required  for  a  traditional  test.  In  addition,  the 
number  of  personnel  participating  in  the  test,  due  solely  to  ADS-specific  requirements,  was 
documented. 

The  ETE  Test  Phase  4  results  did  not  provide  any  set  formula  for  forecasting  personnel  needs. 
Rather,  the  exact  personnel  requirements  for  C4ISR  system  testing  in  an  ADS-enhanced  live  test 
environment  appear  to  vary  from  test  to  test.  With  this  caveat  in  mind  and  using  just  the  ETE 
Test  team  requirements,  an  ADS  environment  does  not  appear  to  add  to  the  personnel  needs  of  a 
C4ISR  test.  Six  JADS  personnel  were  continually  involved  in  ETE  Test  management  and 
planning,  a  number  equivalent  to  the  staff  needed  to  direct  the  day-to-day  operations  of  a 
conventional  C4ISR  test  of  the  same  magnitude.  Thirteen  people  were  involved  in  test 
monitoring  and  data  collection  at  the  various  ETE  Test  nodes  during  the  actual  execution  of  the 
Phase  4  FI,  risk  reduction,  and  operational  tests.  Given  the  distributed  nature  of  C4ISR  systems, 
a  similar  number  of  people  would  be  needed  in  a  comparable,  traditional  C4ISR  test.  Two  N&E 
personnel  were  needed  for  network  support  during  the  execution  of  Phase  4,  and  two  analysts 
handled  the  subsequent  data  analysis  and  reduction.  However,  a  conventional  C4ISR  SUT  would 
probably  need  at  least  two,  and  maybe  more,  network  engineers  because  of  its  distributed  nature, 
since  it  might  not  be  as  reliable  as  the  ETE  Test  Phase  4  network.  In  addition,  the  C4ISR 
environment  would  likewise  require  a  minimum  of  two  data  analysts. 

4.2  JADS  Issue  2.  What  are  the  critical  constraints,  concerns,  and  methodologies 
when  using  ADS  for  T&E? 

The  Phase  4  ETE  Test  demonstrated  that  there  were  no  real  technical  barriers  to  using  an  ADS- 
enhanced  live  test  environment  to  provide  a  realistic  test  environment  for  a  C4ISR  system.  This 
is  due  to  the  high  reliability  of  the  network  architecture  underlying  an  ADS  environment  and  the 
dramatic  increases  in  computer  processing  and  storage  capabilities  over  the  past  few  years. 
Rather,  the  key  constraints  to  ADS-enhanced  testing  are  time  and  money.  How  soon  do  you  need 
results?  How  much  are  you  willing  to  spend  on  development? 

The  following  paragraphs  discuss  constraints  and  concerns  in  detail. 


46 


4.2.1  JADS  Objective  2-1.  Assess  the  critical  constraints  and  concerns  in  ADS 
performance  for  T&E. 

4.2.L1  JADS  Subobjective  2-1-2.  Assess  network  and  communications  performance  constraints 
and  concerns. 

JADS  Measures  2-1-2-2.  Number  of  ADS  trials  canceled  or  otherwise  not  used  because  of 
network  problems. 

JADS  Measures  2-1-3-3.  Number  of  trials  in  which  network  connections  were  lost  long 
enough  to  require  trial  cancellation. 

For  these  measures,  the  network  was  defined  as  including  all  software  and  hardware  used  for 
connecting  the  distributed  sites  and  all  loggers  and  instmmentation  used  for  recording  network 
data.  NIUs  were  considered  part  of  the  individual  simulations  and  not  part  of  the  network. 

For  each  trial,  an  execution  log  was  maintained  at  each  node.  The  data  collectors  annotated  all 
problems  encountered  as  well  as  their  causes.  A  test  controller  log  also  documented  the  overall 
status  of  the  network  and  test  trials.  In  addition,  network  monitoring  tools  were  used  to  monitor 
the  status  of  all  network  links  between  nodes.  Any  problems  detected  by  the  monitoring  tools 
were  documented  via  line  printers  in  terms  of  a  brief  explanation  of  the  problem,  the  time,  and 
the  link(s)  involved. 

The  risk  reduction  efforts  taken  during  the  previous  ETE  Test  phases  helped  to  ensure  the 
reliability  of  the  network  during  Phase  4.  Table  4  shows  the  dates  of  the  Phase  4  trials  and  their 
test  times. 


Table  4.  ETE  Test  Phase  4 


Trial 

Vignette 

Test  Time 

Comments 

15  Mar  99 

3 

N/A 

Trial  canceled 

16  Mar  99 

2 

6  hrs,  58  mins 

17  Mar  99 

3 

7  hrs,  44  mins 

19  Mar  99 

3 

5  hrs,  49  mins 

Live  flight 

24  Mar  99 

3 

7  hrs 

25  Mar  99 

3 

5  hrs,  52  mins 

Live  flight 

29  Mar  99 

4 

7  hrs,  3  mins 

30  Mar  99 

5 

5  hrs,  55  mins 

31  Mar  99 

5 

3  hrs,  48  mins 

Live  flight 

JADS  Measure  2-1-2-3.  Average  and  peak  bandwidth  used  by  link. 

This  measure  provided  an  indication  of  bandwidth  use  and  packet  rate  during  the  OT.  Although 
bandwidth  utilization  was  not  expected  to  exceed  capacity,  the  utilization  rate  was  documented  to 


47 


provide  other  ADS  testers  with  an  indication  of  the  amount  of  needed  bandwidth.  The  packet 
rate  data  are  also  included  because  of  their  potential  value  to  other  ADS  testers. 

Data  were  collected  using  the  SPECTRUM®  network  analysis  tool.  SPECTRUM®  provided  the 
capability  to  study  multiple  aspects  of  network  link  performance  including  packet  rate  and 
percentage  of  bandwidth  utilized.  A  polling  rate  of  fifteen  seconds  was  used  in  the  collection  of 
these  data. 

Once  all  the  data  were  gathered,  the  JADS  analysts  consolidated  the  data  by  network  link.  These 
data  were  then  used  to  calculate  daily  packet  rate  and  bandwidth  values  (maximum  and  average) 
for  each  link.  SPECTRUM®  provided  the  bandwidth  values  as  the  percentage  of  bandwidth 
available  on  the  T-1  line.  A  T-1  line  has  a  normal  bandwidth  of  1.544  megabits  per  second 
(Mbps).  For  the  ETE  Test,  some  of  the  bandwidth  of  the  T-1  line  was  reserved  for  voice  traffic 
leaving  a  maximum  bandwidth  available  of  1.344  Mbps. 

Table  5  shows  average  and  maximum  performance  values  for  the  classified  network  links 
monitored  during  the  eight  days  of  active  ETE  Phase  4  testing. 

Packet  rate  and  bandwidth  utilized  across  the  three  network  links  varied  across  the  eight  days  of 
testing  depending  on  the  scenario  being  run  and  the  availability  of  each  link.  Data  traffic  patterns 
also  differed  between  live  flight  (19,  25,  and  31  March)  and  nonlive  flight  test  days.  The  packet 
rate  experienced  over  the  TCAC-Northrop  Grumman  link  ranged  from  10.6  to  19.8  packets  per 
second,  on  average  utilizing  less  than  1%  of  the  link’s  bandwidth  capacity.  Average  utilization 
of  the  remaining  network  links  was  similarly  low,  as  expected,  since  the  Northrop  Grumman-Fort 
Hood  link  was  not  utilized  to  pass  data  traffic  on  live  flight  days,  and  the  Fort  Hood-TCAC  link 
was  only  used  as  an  alternate  traffic  route  when  one  of  the  other  links  experienced  equipment 
problems  or  outages.  Average  packet  rate  over  the  Northrop  Grumman-Fort  Hood  link  ranged 
from  15.6  to  28.3  packets  per  second  on  nonlive  flight  test  days.  Nontest  related  residual  data 
over  the  Fort  Hood-TCAC  link  resulted  in  a  4  packet  per  second  data  rate.  The  maximum  packet 
rate  for  any  network  link  during  the  eight  test  days  was  345  packets  per  second,  experienced  over 
the  TCAC-Northrop  Grumman  link  on  29  March  1999,  resulting  in  a  peak  load  of  69%  of 
bandwidth  capacity.  The  maximum  packet  rate  over  the  Northrop  Grumman-Fort  Hood  link  was 
93  packets  per  second,  experienced  on  17  March,  a  nonlive  flight  day,  with  a  corresponding  load 
of  4%.  All  atypical  traffic  flow  on  the  Fort  Hood-TCAC  link,  the  greatest  of  which  resulted  in  a 
peak  31  per  second  packet  rate,  was  caused  by  equipment  outages  on  other  network  links,  since 
data  were  automatically  rerouted  across  the  remaining  usable  links.  Occasional  equipment 
problems  involving  any  one  network  link  usually  lasted  no  longer  than  three  minutes  and  were 
transparent  to  the  user,  proving  the  utility  of  network  link  redundancy.  No  test  time  was  lost 
because  of  network  equipment. 
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Table  5.  Link  Performance  Characteristics* 


Day 

Node  A 

Node  B 

Load 

Average  Maximum 

Packet  Rate 

Average  Maximum 

T 

G 

0.5  % 

9%. 

15.9/sec 

60/sec 

16  Mar  99 

G 

H 

0.5  % 

4% 

25.4/sec 

92/sec 

H 

T 

0% 

0% 

2.9/sec 

4/sec 

T 

G 

1.2% 

12% 

19.6/sec 

69/sec 

17  Mar  99 

G 

H 

0.6% 

4% 

25.2/sec 

93/sec 

H 

T 

0% 

0% 

2.9/sec 

3/sec 

T 

G 

0.9% 

^  5  % 

19.8/sec 

55/sec 

19  Mar  99 

G 

H 

0% 

2% 

1.1/sec 

29/sec 

H 

T 

0% 

6% 

3.1/sec 

31/sec 

T 

G 

0.3% 

2% 

11.3/sec 

89/sec 

24  Mar  99 

G 

H 

0% 

1  % 

15.6/sec 

28/sec 

H 

T 

0% 

3  % 

4.1/sec 

19/sec 

T 

G 

0.9  % 

67% 

17.1/sec 

310/sec 

25  Mar  99 

G 

H 

0% 

0%  ^ 

1.1/sec 

15/sec 

H 

T 

0% 

3  % 

4.1/sec 

18/sec 

T 

G 

0.7% 

69% 

14.4/sec 

345/sec 

29  Mar  99 

G 

H 

0.6% 

4% 

28.3/sec 

93/sec 

H 

T 

0.1  % 

2% 

4.2/sec 

19/sec 

T 

G 

0.3  % 

1  % 

10.6/sec 

21 /sec 

30  Mar  99 

G 

H 

0.4% 

4% 

26.5/sec 

92/sec 

H 

T 

0% 

3% 

4/sec 

15/sec 

T 

G 

0.3  % 

2% 

12/sec 

86/sec 

31  Mar  99 

G 

.  H 

0% 

0% 

1/sec 

2/sec 

H 

T 

0% 

0% 

3.9/sec 

4/sec 

T 

G 

0.6% 

69  % 

15.1/sec 

345/sec 

Total 

G 

H 

0.3  % 

4  % 

15.5/sec 

93/sec 

H 

T 

0  % 

6  % 

3.7/sec 

31/sec 

T  =  TCAC  G  =  Northrop  Grumman  H  =  Fort  Hood 


*  Table  refers  only  to  active  test  time  during  which  PDU  loggers  were  recording  data. 
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JADS  Measure  2-1-2-5.  Percentage  of  time  PDUs  were  received  out  of  order  by  a  network 
node. 

JADS  Measure  2-1 -2-6,  Percentage  of  total  PDUs  required  at  a  node  that  were  delivered  to 
that  node. 

JADS  Measure  2-1 -2-7.  Average  and  peak  data  latency  between  ADS  nodes. 

The  flow  of  PDUs  to  and  from  each  node  was  recorded  using  loggers  installed  as  part  of  the 
network  architecture.  The  loggers  specifically  recorded  the  time  and  order  that  the  PDUs  were 
transmitted  and  received  at  each  node. 

The  raw  logger  data  were  transformed  and  reduced  for  analysis  to  determine  out  of  order  PDUs, 
duplicate  PDUs,  lost  PDUs,  and  PDU  latency.  These  data  were  then  used  to  calculate  the 
percentage  of  out  of  order,  duplicate,  and  lost  PDUs  at  each  node  for  each  vignette  and  for  the 
test  as  a  whole.  The  minimum,  maximum,  and  mean  latency  of  PDUs  were  also  computed. 
JADS  analysts  accomplished  these  calculations  using  UNIX®-based  software  tools  created  by 
JADS  programmers. 

Tables  6  and  7  describe  the  results  for  these  measures.  Table  6  shows  the  PDU  data  for  each 
vignette  by  node;  there  were  no  duplicate  or  out  of  order  PDUs.  Table  7  provides  the  latency 
data  for  the  vignettes  broken  down  into  the  individual  network  links  between  nodes.  These 
tables  indicate  the  high  reliability  of  the  network  in  passing  PDUs  and  the  network  ability  to 
maintain  stable  latencies  during  the  Phase  4  test. 


Table  6.  Vignette  PDU  Data 


Day 

Node  A 

NodeB 

PDUs  Sent 

PDUs  Received 

PDUs  Lost/ 
Percent  Lost 

16  Mar  99 

W 

T 

91,117 

91,088 

29 

0.03% 

T 

G 

91,088 

90,096 

992 

1.09% 

S 

W 

4,695 

4,684 

11 

0.23% 

17  Mar  99 

W 

T 

167,396 

165,824 

1,572 

0.94% 

T 

G 

165,824 

165,711 

113 

0.07% 

S 

W 

4,679 

4,679 

0 

0% 

19  Mar  99 
(Flight) 

w 

T 

122,211 

120,724 

1,487 

1.25% 

T 

G 

120,724 

120,618 

106 

0.09% 

S 

W 

2,998 

2,998 

0 

0% 

NIU 

E-8C 

120,618 

120,190 

428 

0.35% 
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Table  6.  Vignette  PDU  Data  (cent.) 


Day 

Node  A 

NodeB 

PDUs  Sent 

PDUs  Received 

PDUs  Lost/ 

Percent  Lost 

W 

T 

142,332 

140,977 

1,355 

0.95% 

24  Mai-  99 

T 

G 

140,977 

139,939 

1,038 

0.74% 

S 

W 

3,709 

3,709 

0 

0% 

w 

T 

131,559 

130,581 

978 

0.74% 

25  Mar  99 
(Flight) 

T 

G 

130,581 

130,025* 

556* 

0.43% 

S 

W 

3,473 

3,473 

0 

0% 

NIU 

E-8C 

130,025 

129,798 

227 

0.17% 

29  Mar  99 

W 

T 

100,477 

100,367 

no 

0.11% 

T 

G 

100,367 

99,893* 

474* 

0.47% 

S 

W 

4,673 

4,672 

1 

0.02% 

30  Mar  99 

w 

T 

111,391 

111,094 

297 

0.27% 

T 

G 

111,094 

110,933 

161 

0.14% 

S 

W 

3,958 

3,958 

0 

0% 

31  Mar  99 
(Flight) 

W 

T 

86,973 

86,787 

186 

0.21% 

T 

G 

86,787 

86,787 

0 

0% 

S 

W 

1,904 

1,904 

0 

0% 

NIU 

E-8C 

86,787 

77,595 

9,192 

10.59% 

Total 

W 

T 

953,456 

947,442 

6,014 

0.63% 

T 

G 

947,442 

944,002 

3,440 

0.36% 

S 

W 

30,089 

30,077 

12 

0.04% 

NIU 

E-8C 

359,144 

349,297 

9,847 

2.74% 

W 

=  WSMR  T  =  TCAC  G  =  Norl 

:hrop  Grumman  S  =  Fort  Sill 

*  For  the  25  and  29  March  tests,  the  numbers  indicated  for  PDUs  received  were  taken  from 
the  loggers.  However,  analysis  of  the  Northrop  Grumman  log  files  for  25  and  29  March 
showed  much  lower  PDU  numbers  at  the  Northrop  Grumman  node  (108,020  and  91,119, 
respectively).  These  PDU  losses  resulted  from  computer  work  done  at  the  Northrop 
Grumman  node  during  test  days.  The  work  was  stopped  following  the  29  March  test,  and  the 
problem  was  resolved. 
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Table  7.  Vignette  Latency  Data 


Day 

Node  A 

NodeB 

Latency  (seconds)  I 

Minimum 

Maximum 

Mean 

16  Mar  99 

W 

T 

0.017 

0.131 

0.039 

T 

G 

0.048 

0.280 

0.053 

S 

W 

0.039 

0.348 

0.038 

17  Mar  99 

w 

T 

0.019 

0.172 

0.052 

T 

G 

0.048 

0.279 

0.059 

S 

W 

0.036 

0.373 

0.038 

19  Mar  99 
(Flight) 

W 

T 

0.020 

0.162 

0.052 

T 

G 

0.042 

0.169 

0.061 

S 

W 

0.037 

0.370 

0.038 

NIU 

E-8C 

1.58 

41.68 

12.08 

24  Mar  99 

W 

T 

0.020 

0.173 

0.050 

T 

G 

0.039 

0.186 

0.060 

S 

W 

0.035 

0.379 

0.037 

25  Mar  99 
(Flight) 

W 

T 

0.018 

0.178 

0.047 

T 

G 

0.047 

0.291 

0.059 

S 

W 

0.035 

0.376 

0.038 

NIU 

E-8C 

2.82 

29.95 

12.99 

29  Mar  99 

W 

T 

0.019 

0.156 

0.046 

T 

G 

0.048 

0.289 

0.058 

S 

W 

0.037 

0.395 

0.038 

30  Mar  99 

W 

T 

0.020 

0.161 

0.044 

T 

G 

0.041 

0.224 

0.056 

S 

W 

0.035 

0.385 

0.038 

31  Mar  99 
(Flight) 

W 

T 

0.018 

0.157 

0.041 

T 

G 

0.045 

0.177 

0.055 

S 

W 

0.036 

0.399 

0.035 

NIU 

E-8C 

2.06 

85.57 

12.60 

Total 

W 

T 

0.017 

0.178 

0.046 

T 

G 

0.039 

0.291 

0.058 

S 

W 

0.035 

0.399 

0.038 

NIU 

E-8C 

1.58 

85.57 

12.56 

W=  WSMR  T  =  TCAC  G  =  Northrop  Grumman  S  =  Fort  Sill 


Tables  6  and  7  show  the  reliability  of  the  network  in  passing  PDUs  and  maintaining  low  latencies 
during  the  Phase  4  risk  reduction  and  OT.  The  PDU  data  in  Table  6  show  total  PDU  loss  rates  of 
0.63,  0.36,  and  0.04  percent  for  the  WSMR-TCAC,  TCAC-Northrop  Grumman,  and  Fort  Sill- 
WSMR  links  respectively.  These  percentages  equate  to  an  overall  PDU  loss  rate  of  0.49%.  This 
PDU  loss  rate  is  much  lower  than  the  loss  rate  experienced  during  the  Phase  3  test  (1.63%)  and  is 
well  below  the  criterion  of  5%  set  for  overall  PDU  losses.  This  suggests  a  high  level  of 
reliability  was  experienced  during  the  test  events. 
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The  latency  data  in  Table  7  show  that  the  average  latency  for  the  TCAC-Northrop  Grumman  and 
Fort  Sill  -  WSMR  links  was  very  stable  over  the  eight  days  of  testing,  varying  by  less  than  10%. 
However,  the  WSMR-TCAC  link  exhibited  relatively  large  variations  in  latency  (up  to  33%). 
The  latency  over  the  TCAC-Northrop  Gmmman  link  had  maximum  values  of  almost  400 
milliseconds.  However,  these  were  only  transient  values  and  did  not  significantly  affect  the 
average  latency  of  the  links.  The  average  total  latency  for  the  ESPDU  data  to  travel  from  the 
WSMR  node  to  the  Northrop  Grumman  node  was  just  over  100  milliseconds  and  had  no  effect 
on  the  validity  of  the  Phase  4  results. 

Tables  6  and  7  also  show  the  performance  of  the  SATCOM  link  during  the  three  live  flights. 
The  PDU  data  in  Table  6  show  PDU  loss  rates  of  0.35%,  0.17%,  and  10.59%  during  the  19,  25 
and  31  March  live  flights  respectively.  These  rates  equate  to  an  overall  PDU  loss  rate  of  2.74% 
for  the  SATCOM  link.  The  PDU  loss  rates  for  the  19  and  25  March  live  flights  are  comparable 
to  those  of  the  network  links,  suggesting  that  the  SATCOM  link  was  highly  reliable  during  these 
test  dates.  However,  the  high  PDU  loss  rate  experienced  on  31  March  suggests  that  there  may 
have  been  problems  in  passing  data  via  the  SATCOM  link  for  this  trial. 

During  Phase  4  flight  tests  on  19,  25  and  31  March,  observers  noted  and  analysis  confirmed  that 
small  groups  of  entities  (1-15)  were  traveling  in  straight  lines  across  the  GRCA  without  regard  to 
terrain  features  such  as  rivers,  lakes  or  roads.  Some  of  the  entities  would  continue  to  move  until 
the  conclusion  of  the  vignette  while  others  would  disappear. 

The  impact  of  the  wandering  entities  was  noted  during  the  targeting  phase  at  Fort  Hood.  When 
the  TAC  personnel  conferred  with  the  Janus  operator  prior  to  fire  missions,  it  was  noted  the 
wandering  entities  were  not  in  that  location  within  the  Janus  simulation.  The  entities  were  seen  5 
to  25  km  from  their  expected  locations.  When  fire  missions  were  directed  at  those  targets  as 
displayed  in  the  GSM  no  hits  were  seen  in  Janus.  As  a  result  of  these  observations  further 
analysis  was  conducted.  Analysts  replayed  the  data  tapes  recorded  in  the  GSM  and  compared 
them  with  test  logs  from  all  nodes  during  the  three  live  flights.  The  GSM  tapes  were  studied  for 
apparent  entities  that  “wandered.”  The  Zulu  times  of  these  anomalies  were  compared  to  events 
that  were  occurring  at  other  nodes. 

The  analysis  revealed  three  apparent  causes  for  these  wandering  entities.  The  first  and  most 
notable  event  resulted  from  software  crashes  of  the  GNIU.  When  the  GNIU  is  inoperable  and 
not  passing  VSTARS  data  packets  (VDPs),  the  entities  on  board  the  E-8C  will  continue  to  dead 
reckon  on  their  last  received  velocity.  If  an  entity  changes  velocity  during  the  GNIU  outage,  it 
will  not  be  passed  to  the  E-8C.  There  were  some  cases  noted  where  entities  stopped  in  the  Janus 
vignette  during  a  GNIU  outage,  and  as  a  result,  those  entities  on  the'  E-8C  were  continued  on 
their  last  course  and  speed  for  the  remainder  of  the  vignette.  It  was  further  observed  that  other 
entities  which  changed  state  during  GNIU  outages  continued  on  the  last  course  and  speed  until 
they  received  the  next  change  of  state  from  the  Janus  vignette.  These  entities  that  appeared  to 
miss  turns  during  the  outages  “snapped”  back  when  the  next  ESPDU  from  the  vignette  was 
received.  Although  the  GNIU  outages  were  not  frequent  or  lengthy,  they  resulted  in  the  most 
notable  wandering  if  they  occurred  during  heavy  activity  (change  of  state)  within  the  Janus 
vignettes. 
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The  second  cause  was  loss  of  VDPs  being  transmitted  to  the  aircraft  from  the  GNIU.  Losses 
varied  in  how  many  and  how  often  they  occurred.  The  losses  varied  from  one  lost  VDP  to  850 
lost  during  a  single  occurrence.  The  vast  majority  of  the  cases  were  only  several  lost  VDPs  at  a 
time.  Table  6  lists  the  total  loss  of  VDPs  for  each  of  the  flight  days.  Monitoring  tools  were  not 
in  place  to  determine  the  exact  cause  of  the  losses,  but  they  could  have  resulted  from 
transmission  losses  from  the  GNIU  to  the  satellite  to  reception  losses  on  the  aircraft  during  turns. 
The  majority  of  these  losses  would  only  result  in  isolated  entities  wandering  off  across  the 
GRCA. 

The  third  cause  was  from  PDU  losses  over  the  WAN.  Table  6  lists  the  number  of  PDUs  lost 
during  each  test  event.  These  losses  produced  the  same  symptoms  as  seen  from  GNIU  crashes  or 
SATCOM  losses.  The  WAN  losses  were  minimal  and  resulted  in  the  wandering  and  snapping 
seen  from  other  causes. 

The  overall  impact  on  the  Phase  4  test  objectives  from  these  losses  was  minimal.  Several 
methods  could  be  used  to  reduce  or  eliminate  these  type  of  problems  in  future  tests  with  a  similar 
design.  The  Phase  4  ETE  test  design  was  frozen  after  the  start  of  the  phase  and  refinements  were 
not  allowed  to  the  hardware  or  software  to  improve  the  reliability  or  performance  of  individual 
systems.  The  impact  of  GNIU  crashes,  which  were  infrequent,  as  well  as  the  WAN  and 
SATCOM  losses,  could  be  reduced  by  running  a  simulation  with  less  moving  entities  or  using  a 
heartbeat  to  update  all  entities  on  a  regular  basis.  The  heartbeat  would  “catch”  entities  that 
missed  a  change  of  state  during  an  outage.  The  heartbeat  during  the  Phase  4  test  was  turned  off 
at  Janus  to  minimize  the  VDP  load  on  the  SATCOM  link.  The  SATCOM  bandwidth  used  for 
this  test  was  19.2  kilobits  per  second  (Kbits),  and  it  was  the  limiting  factor  in  the  transmission  of 
data.  The  use  of  a  more  robust  communication  link  would  allow  for  the  use  of  a  heartbeat  even 
with  a  large  number  of  movers. 

Table  7  shows  the  latencies  experienced  on  the  SATCOM  link  during -the  live  flights.  These 
latency  numbers  are  much  higher  than  those  experienced  on  the  network  links:  latencies  as  high 
as  85  seconds  were  experienced  on  31  March. 

Further  investigation  revealed  that  these  times  were,  in  fact,  not  latencies,  but  rather  delays  in 
logging  the  arrival  of  the  data  on  board  the  aircraft.  The  aircraft  used  its  general  purpose 
computers  (GPC)  to  log  all  local  area  network  (LAN)  traffic  that  occurred.  These  same  GPC 
also  ran  the  primary  radar  software  and  were  very  busy.  Logging  on  these  computers  is  a 
secondary  task  and  is  often  delayed  for  long  periods  of  time  (minutes).  During  the  test,  no 
computers  were  available  on  the  LAN  to  dedicate  to  logging,  and  JADS  was  not  allowed  to  add  a 
dedicated  logger  on  board  the  aircraft. 

The  smallest  delay  in  logging  observed  for  the  satellite  link  was  on  the  order  of  2  seconds.  If  one 
assumes  that  these  instances  occurred  during  periods  of  minimal  load  on  the  GPCs,  than  the 
latency  from  the  GNIU  to  the  ANIU  is  no  more  than  2  seconds  and  would  have  little  effect  on  the 
operators.  This  assumption  was  supported  both  by  observation  during  the  flights  and  by 
calculations  showing  that  the  total  transmission  and  processing  times  were  less  than  2  seconds. 
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Future  tests  using  the  actual  SUT  must  come  to  grips  with  data  logging  using  nondedicated 
computers.  The  Joint  STARS  JTF,  who  regularly  tests  Joint  STARS,  could  accept  the  delays  in 
logging  because  its  data  are  not  time  sensitive.  JADS  could  not  but  was  prevented  from 
modifying  the  SUT.  Testers  must  identify  early  in  a  program’s  life  cycle  not  only  the  data  that 
will  be  collected  during  testing  but  also  data  that  are  time  sensitive  as  to  when  they  are  collected. 

JADS  Measure  2-1-3-6.  Average  downtime  due  to  ADS  network  failures. 

This  measure  identified  the  impact  of  network  failures  on  the  OT.  The  network  system  included 
the  local  area  networks  at  the  TCAC  and  Northrop  Grumman,  as  well  as  the  wide  area  network 
connecting  WSMR  to  the  TCAC,  the  TCAC  to  Northrop  Grumman,  Fort  Sill  to  the  TCAC,  and 
Fort  Hood  to  the  TCAC.  During  Phase  4,  logs  were  kept  to  record  all  network  problems,  the 
start  time  and  duration  of  the  problems,  and  problem  resolution.  In  addition,  network  monitoring 
tools  were  used  to  monitor  the  status  of  all  network  links  between  the  nodes.  Any  problem 
detected  by  the  monitoring  tools  was  documented  via  line  printers  in  terms  of  a  brief  explanation 
of  the  problem,  the  time,  and  the  link(s)  involved. 

All  the  trials  executed  during  the  Phase  4  test  were  successful  with  PDU  losses  not  exceeding 
five  percent  for  any  trial.  For  the  eight  trials  that  were  accomplished,  16  network  outages  were 
experienced  resulting  in  a  total  of  39  minutes  of  network  downtime.  Table  8  shows  the  data  on 
network  downtime;  Table  9  provides  data  on  the  impact  of  network  downtime  on  PDU  losses. 
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Table  8.  Network  Downtime 


Day 

Duration  of 
Trial 

16  Mar  99 

6  hrs,  58  mins 

17  Mar  99 

7  hrs,  44  mins 

19  Mar  99 

5  hrs,  49  mins 

24  Mar  99 

7  hrs 

25  Mar  99 

5  hrs,  52  mins 

29  Mar  99 

7  hrs,  3  mins 

30  Mar  99 

5  hrs,  55  mins 

31  Mar  99 

3  hrs,  48  mins 

2  mins 


None 
2  mins 

7  mins 


14  mins 


12  mins 


2  mins 
None 


%  of  Time 

Network  Reason  UnavaUable 
le  Unavailable 

r 

0.48%  Router  down  at 

_ _ _ Northrop  Grumman 

_ None _ Not  applicable _ 

0.57%  Trunk  card  down  at 

_ TCAC _ 

1 .67  %  Router  down  at 

Northrop  Grumman; 
Northrop  Grumman 

_ port  deactivated  twice 

3.98%  Trunk  card  down  three 

times  at  TCAC; 
Northrop  Grumman 
port  deactivated;  router 
down  twice  at  Northrop 

_ Grumman _ 

2.84%  Trunk  card  down  three 

times  at  TCAC;  router 
down  at  Northrop 

_ _ Grumman _ 

0.56%  Trunk  card  down  at 

_ TCAC _ 

 None  Not  applicable 


Total 


50  hrs,  9  mins 


39  mins 


1.30% 


Table  9.  Analysis  of  Network  Impact  on  PDUs 


16  Mar  99 


17  Mar  99 


19  Mar  99 


24  Mar  99 


Node  Node 
A  B 


25  Mar  99 


PDUs 

lost 


91,117 


29 


91,088 


11 


167,396 


1,572 


165,824 


0 


122,211 


1,487 


120,724 


0 


142,332 


1,355 


140,977 


0 


131,559 


978 


130,581 


Network-related  PDU  losses 


%  of  PDUs  /  %  of  PDUs 
sent  lost 


0%/0% 


952 

1.05%  /  95.97% 


0 


0%/0% 


0 


0%/0% 


0 


0%/0% 


0 


0%/0% 


0 


0%/0% 


75 


0.06%  /  70.75% 


0 


0%/0% 


0 


0%/0% 


604 


0.43%  /  58.19% 


0 


0%/0% 


0 


0%/0% 


510 


0.39%  791.73% 


0 


0%/0% 


Unexplained  PDU  losses 


%  of  PDUs /%  of  PDUs 
sent  lost 


29 


0.03%  / 100% 


40 


0.04%  74.03% 


11 


0.23%  7  100% 


1,572 


0.94%  7 100% 


113 


0.06%  7  100% 


0 


0%70% 


1,487 


1.22%  7 100% 


31 


0.03%  7  29.95% 


0 


0%70% 


1,355 


0.95%  7 100% 


434 


0.31%  741.81% 


0 


0%  7  0% 


978 


0.74%  7 100% 


46 


0.04%  7  8.27% 


0 


0%70% 


Table  9.  Analysis  of  Network  Impact  on  PDUs  (cont.) 


Day 

Node 

A 

Node 

B 

PDUs 

sent 

Network-related  PDU  losses 

%  of  PDUs  /%  of  PDUs 
sent  lost 

Unexplained  PDU  losses 

PDUs 

lost 

%  of  PDUs  /  %  of  PDUs 
sent  lost 

29  Mar  99 

W 

T 

100,477 

0 

110 

110 

me  10% 

0.11%  7  100% 

T 

G 

100,367 

432 

42 

474 

0.43%  791.14% 

0.04%  7  8.86% 

S 

W 

4,673 

0 

1 

1 

0%/0% 

0.02%  7  100% 

30  Mar  99 

W 

T 

111,391 

0 

297 

297 

0%/0% 

0.27%  7  100% 

T 

G 

111,094 

157 

4 

161 

0.14%  7  97.52% 

0.004%  7  2.48% 

S 

W 

3,958 

0 

0 

0 

0%70% 

0%70% 

w 

T 

86,973 

0 

186 

186 

0%  7  0% 

0.21%  7  100% 

31  Mar  99 

T 

G 

86,787 

0 

6 

0 

0%70% 

0%70% 

S 

W 

1,904 

0 

0 

0 

0%70% 

0%70% 

Total 

w 

T 

953,456 

0 

6,014 

6,014 

0%70% 

0.63%  7  100% 

T 

G 

947,442 

2,730 

710 

3,440 

0.29%  7  79.36% 

0.07%  7  20.64% 

S 

w 

30,089 

0 

•  12 

12 

0%70% 

0.04%  7  100% 

W  -  WSMR  T  =  TCAC  G  =  Northrop  Grumman  S  =  Fort  Sill 

Tables  8  and  9  show  that  the  network  was  reliable  during  the  execution  of  the  Phase  4  test. 
During  the  eight  trials,  the  network  was  down  for  only  39  minutes  or  1.30%  of  the  test  period, 
resulting  in  relatively  few  lost  PDUs.  All  the  network  problems  were  minor  with  the  majority 
caused  by  TCAC  trunk  card  outages  or  the  Northrop  Grumman  router.  In  all,  16  periods  of 
network  downtime  were  experienced  with  each  outage  lasting  between  1  and  3  minutes.  These 


58 


results  are  comparable  to  the  Phase  3  test,  which  experienced  1.29%  network  downtime.  The 
Phase  4  test  results  also  compare  favorably  to  the  Phase  3  results  in  terms  of  the  impact  of  the 
network  on  PDU  losses.  While  the  Phase  3  test  experienced  network-related  PDU  loss  rates  of 
2.13%,  0.40%,  and  1.58%  for  the  three  test  links,  no  Phase  4  PDU  loss  rate  exceeded  0.30%  due 
to  the  impact  of  network  downtimes. 

There  are  two  possible  explanations  for  a  “lost”  PDU.  One  is  that  the  network  lost  the  PDU 
either  through  an  outage  or  because  the  router  dropped  the  PDU.  The  other  explanation  could  be 
that  the  logger  failed  to  log  the  PDU.  One  path  of  the  WAN  had  four  loggers  and  used  four 
routers.  As  one  proceeded  down  the  path  from  router  to  router,  each  successive  logger  logged  a 
lower  number  of  PDUs.  A  downstream  logger  never  logged  more  PDUs.  than  the  preceding 
logger.  This  would  indicate  that  the  PDUs  were  lost  by  the  routers. 

4.2.L2  JADS  Subobjective  2-1-3.  Assess  the  impact  of  ADS  reliability,  availability,  and 
maintainability  on  T&E. 

JADS  Measures  2-1-3-1.  Percentage  of  trials  delayed,  rescheduled,  and/or  redone  because 
of  ADS  systems  (exclusive  of  network)  unavailability. 

JADS  Measures  2-1-3-5.  Mean  operating  time  between  ADS  system  failures  (severe  enough 
to  require  trial  cancellation). 

These  measures  determined  the  availability  of  ADS  nodes,  including  the  NIUs,  and  the  impact  of 
node  failures  on  Phase  4  testing. 

For  each  trial,  an  execution  log  was  maintained  at  each  node.  The  data  collectors  annotated  all 
problems  encountered  with  the  ADS  systems  along  with  their  causes.  A  test  controller  log  was 
also  maintained  to  document  the  overall  status  of  the  trials. 

Exclusive  of  network  unavailability,  one  trial  had  to  be  canceled  and  reaccomplished  during  the 
Phase  4  test.  This  cancellation  resulted  from  a  major  ADS  system  failure,  which  could  not  be 
resolved  during  the  scheduled  time  for  the  trial.  Table  10  lists  this  trial  cancellation  along  with 
the  other  ADS  system  failures  experienced  during  Phase  4.  The  table  also  includes  the  time 
required  to  resolve  these  failures  as  well  as  their  impact. 
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Table  10.  ADS  System  Failures 


Day 


Failure 


Resolution 


Duration 


15  Mar  99 


SCDL  down; 
vignette  on 
tape  at  Fort 
Hood 


N/A 


N/A 


16  Mar  99 


No  ADS 

system 

failures 


N/A 


N/A 


17  Mar  99 


Entities 
straying  from 
roads  on 
VSTARS 


VSTARS 
crashed  due  to 
disk  usage 
problem _ 


VSTARS 
brought  down 
for 

adjustments 


Disk  usage 
examined; 
VSTARS 
rebooted 


25  mins 


1  hr,  42 
mins 


19  Mar  99 


No  ADS 

system 

failures 


N/A 


N/A 


24  Mar  99 


25  Mar  99 


VSTARS 
crash  due  to 
SAR  SIM 


VSTARS 

rebooted 


VSTARS 
crash  due  to 
continuing 
SAR  SIM 
problems 


VSTARS 

rebooted 


Time  problem 
(Fort  Hood  to 
Fort  Sill)  due 
to  ASAS 
machine 


Time  updated 
on  ASAS 
machine  at 
Fort  Hood 


AFATDS  at 
Fort  Hood 
crashed 


AFATDS 

rebooted 


MTI  SIM 
crashed  on 
E-8C 


Problem  not 
resolved  while 
on  station 


17  mins 


45  mins 


15  mins 


15  mins 


1  hr,  2  mins 


29  Mar  99 


SAR  SIM  not 
working  at 
Northrop 
Grumman 


Hardware 
problem  found 
and  resolved 


1  hr,  25 
mins 


30  Mar  99 


No  ADS 

system 

failures 


N/A 


N/A 


31  Mar  99 


MTI  SIM 
crashed  on 
E-8C 


Test  Time 

Impact  on  Test 

N/A 

Trial  canceled 

6  hrs,  58 
mins 

N/A 

7  hrs,  44 
mins 

Trial  paused  while  VSTARS 
was  down 

Trial  paused  while  VSTARS 
was  down 

5  hrs,  49 
mins 

N/A 

7  hrs 

Trial  paused  while  VSTARS 
was  down 

Trial  paused  while  VSTARS 
was  down 

No  fire  missions  executed 
until  time  problem  resolved 

5  hrs,  52 
mins 

Some  fire  missions  lost;  fire 
missions  stopped  until 
problem  resolved 

One  hour  of 
time-on-station  data 
collection  was  lost 

7  hrs,  3 
mins 

Degraded  imagery  provided 
to  Fort  Hood 

5  hrs,  55 
mins 

N/A 

3  hrs,  48 

Ten  minutes  of 

Problem  not 
resolved  while 
on  station 


10  mins 


mins 


Ten  minutes  of 
time-on-station  data 
collection  were  lost 


Extensive  risk  reduction  testing  was  instrumental  in  preventing  any  major  ADS  system  failures 
from  affecting  the  Phase  4  test.  A  total  of  ten  ADS  system  failures  occurred  during  the  nine  days 
of  testing.  One  trial  was  canceled  because  of  ADS  failures,  i.e.,  the  failure  to  establish  SCDL  or 
run  a  taped  vignette  from  Fort  Hood  on  15  March.  In  addition  to  this  cancellation,  there  were 
three  trials  that  were  delayed  for  more  than  an  hour  because  of  ADS  system  failures.  One  trial 
during  the  risk  reduction  portion  of  Phase  4  and  one  trial  during  the  OT  were  delayed  for  more 
than  an  hour  because  of  VSTARS  crashes.  An  MTI  simulation  crash  on  the  E-8C  accounted  for 
the  other  lengthy  ADS  system  delay.  These  three  failures,  along  with  the  remaining  ADS  system 
failures,  accounted  for  a  total  of  6  hours  46  minutes  of  test  time. 

4.2.2  JADS  Objective  2-2.  Assess  the  critical  constraints  and  concerns  in  ADS  support 
systems  for  T&E. 

4.2.2. 1  JADS  Subobjective  2-2-1.  Assess  the  critical  constraints  and  concerns  regarding  ADS 
data  management  and  analysis  systems. 

JADS  Measure  2-2-1-1.  Degree  to  which  ADS  nodes  provide  for  collection,  data  entry,  and 
quality  checking  of  pre-  and  post-trial  briefing  data. 

Quick-look  analysis  of  results  was  used  to  support  the  post-trial  briefings.  This  analysis  relied 
primarily  on  automated  data  collection  at  all  ETE  Test  nodes.  The  data  collection  tools  included 
the  JADS  logger,  which  collected  the  PDU  log  files,  and  a  SPECTRUM®  logger  to  monitor 
network  performance.  Data  collection  tools  were  attached  to  the  network  at  each  node  without 
any  impact  on  network  or  node  performance.  At  the  end  of  each  test  day,  the  data  were  remotely 
retrieved  by  the  TCAC  and  the  file  size  checked.  This  procedure  supported  timely  quick-look 
analysis  and  test  feedback. 

In  addition  to  electronic  data  logs,  manually  written  logs  were  kept  at  each  test  site  and  used  to 
support  post-trial  briefings.  During  the  Phase  4  test,  a  daily  after-action  teleconference  call  was 
conducted.  This  enabled  the  test  controller  to  discuss  and  fully  understand  the  problems  of  the 
day  without  having  to  review  local  log  sheets. 

As  mentioned  previously,  no  instrumentation  of  the  SUT  was  allowed.  This  caused  the 
previously  discussed  problem  with  the  post-test  measurement  of  satellite  link  latencies. 
However,  test  observers  were  unable  to  observe  any  latency  effects  during  the  conduct  of  the  test. 
This  would  indicate  that  the  latency  involved  had  no  effect  upon  the  test. 

JADS  Measure  2-2-1-2.  Adequacy  of  relevant  test  data  storage  at  ADS  nodes. 

The  ETE  Test  analysis  requirements  drove  test  data  storage  needs.  The  focus  of  data  analysis  at 
each  site  was  on  network  latency  as  well  as  on  the  actual  PDU  input  or  data  output  at  each  site. 
The  need  to  record  PDU  traffic  at  each  node  required  a  determination  of  the  data  output  and 
reception  rates  at  all  sites.  The  largest  contributor  to  ESPDU  traffic  was  the  output  of  the  Janus 
simulation.  ESPDUs  from  Janus  are  a  function  of  the  Janus  heartbeat  and  the  vignette  design. 
During  the  Phase  2  testing,  the  Janus  heartbeat  was  set  to  update  all  entities  every  fifteen  minutes 
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during  the  first  hour.  In  addition,  Janus  had  to  output  an  ESPDU  when  an  entity  changed  state, 
i.e.,  start,  stop,  turn,  etc.  As  a  result,  the  ESPDU  output  grew  as  the  number  of  movers  increased. 
The  ETE  Test  was  planned  with  the  use  of  five  different  vignettes  ranging  from  prehostility  with 
a  low  number  of  movers  to  an  active  battle  vignette  with  more  than  3,000  entities  moving  at  one 
time.  Prior  to  Phase  4  testing,  the  five  vignettes  were  played  and  the  ESPDU  output  recorded. 
The  maximum  file  size  seen  during  this  testing  was  about  fifteen  megab34es.  To  support  the  data 
recording  as  well  as  file  storage  and  local  software  requirements,  the  JADS  Network  and 
Engineering  team  installed  4-gigabyte  hard  drives  on  the  SGI  Indy  at  each  node. 

During  preparations  for  the  Phase  4  test,  the  Northrop  Grumman  node  required  the  largest  data 
capacity  in  order  to  support  VSTARS  software  testing  in  a  stand-alone  mode.  This  testing 
required  the  playback  of  PDU  files  recorded  from  TRAC-WSMR  to  VSTARS.  All  five  vignettes 
were  played  back  at  various  times,  and  at  least  five  vignette  PDU  files  were  stored  on  the  SGI 
Indy  at  all  times.  During  actual  Phase  4  testing,  the  ETE  Test  team  found  hard  drive  data  storage 
capacity  to  be  more  than  adequate. 

At  one  point  during  the  development  of  the  test,  a  desire  to  record  ground  truth  data  on  each 
entity  was  expressed.  These  data  would  be  used  to  determine  the  performance  of  the  virtual 
radar  during  the  flight  in  the  same  manner  that  time-space-position  information  (TSPI)  data  are 
used  to  determine  the  actual  radar’s  performance  during  a  flight.  It  was  quickly  determined  that 
the  Joint  STARS  JTF  would  be  unable  to  process  and  sort  out  the  data  on  10,000  entities  and  the 
request  was  cancelled.  Recording  of  these  data,  however,  would  have  required  the  addition  of 
additional  disk  capacity  to  the  aircraft,  an  action  we  were  prevented  from  doing. 

The  development  of  data  storage  needs  required  a  full  understanding  of  each  node’s 
requirements.  Since  the  cost  of  hard  drive  storage  has  decreased  dramatically  over  the  past  few 
years,  it  was  cost  effective  to  allow  for  unexpected  growth  by  significantly  exceeding  the 
expected  storage  requirements. 

JADS  Measure  2-2-l“4.  Ease  with  which  data  can  be  retrieved  post  trial  from  a  given  node. 

During  Phase  4  of  the  ETE  Test,  automated  data  were  collected  using  PDU  loggers  at  the  nodes, 
and  operational  data  were  collected  using  log  sheets  at  each  node.  Automated  data  on  the 
SATCOM  link  were  collected  using  logging  software  developed  by  Northrop  Grumman.  The 
automated  data  collected  at  the  other  JADS  test  locations  during  the  test  were  retrieved  by  the 
TCAC  using  file  transfer  protocol  (FTP),  while  JADS  personnel  transported  the  log  sheets  to 
JADS.  The  automated  data  were  compressed  and  converted  for  analysis  using  JADS  UNDC®- 
based  tools. 

The  automated  data  retrieval  process  was  very  effective.  For  the  ETE  Test  Phase  4,  the 
automated  data  retrieval  process  needed  about  thirty  minutes  despite  the  large  size  of  the  log  files 
being  converted.  Also,  there  were  no  problems  encountered  during  any  retrieval  efforts.  As 
exemplified  by  the  ETE  Test  team’s  data  retrieval  process,  any  ADS  data  retrieval  methodology 
need  not  be  complex. 
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4.2.22  JADS  Subobjective  2-2-2.  Assess  the  critical  constraints  and  concerns  regarding 
conjiguration  management  of  ADS  test  assets. 

JADS  Measure  2-2-2-1.  Degree  to  which  test  managers  can  control  the  configurations  of 
ADS  participants,  the  ADS  environment  data,  and  ADS  networks. 

This  measure  determined  if  the  test  manager  could  adequately  control  the  test  eonfiguration  of 
ADS  participants,  the  ADS  environment  data,  and  the  ADS  network  during  and  between  test 
events.  The  JADS  analysts  conducted  interviews  with  the  test  team  members  involved  in 
configuration  management  on  the  Phase  4  test.  The  JADS  analysts  also  monitored  the  progress 
of  test  and  network  configuration  from  formal  integrated  product  team  (IPT)  and  requirements 
meetings. 

Configuration  control  for  the  ETE  Test  synthetic  environment  proved  to  be  a  challenge.  The 
distributed  nature  of  the  test  made  configuration  control  more  complex  because  of  the  many 
different  organizations  involved  in  the  test.  Unlike  a  single-site  test  effort,  the  ETE  Test 
involved  more  than  half  a  dozen  organizations  and  two  branches  of  the  military.  The  added 
difficulty  of  the  distributed  network  made  frequent  technical  meetings  and  formal  IPTs  a  must. 
These  meetings  provided  the  test  manager  with  the  ability  to  track  progress  and  problems  with 
the  network  configuration  and  data  formats.  These  meetings  also  provided  the  forum  for  the 
system  experts  to  resolve  the  issues  with  all  those  who  were  affected. 

This  process  proved  to  be  effective  in  controlling  the  Phase  4  configuration  during  the  test  build¬ 
up  and  preparation  phase.  In  addition,  there  were  no  configuration  control  problems  during  the 
OT  execution. 

Configuration  control  of  an  ADS  test  can  add  significant  management  issues  when  compared  to 
non-ADS  efforts.  This  added  complexity  requires  a  higher  level  of  involvement  by  the  test 
manager  as  well  as  more  frequent  face-to-face  meetings  among  technical  personnel  from  the 
various  ADS  test  nodes.  An  ADS  test  manager’s  job  would  be  eased  with  the  aid  of  an 
integration  engineer. 

4.2.3  JADS  Objective  2-3.  Develop  and  assess  methodologies  associated  with  ADS  for 
T&E. 

4.2.3. 1  JADS  Subobjective  2-3-2.  Develop  and  assess  methodologies  associated  with  test 
execution  and  control  for  tests  using  ADS. 

JADS  Measure  1-2-2-3.  Degree  to  which  test  exercise/control  features  can  be  improved 
through  experimentation  using  an  ADS  environment. 

JADS  Measure  2-3-2-3.  Degree  to  which  protocols,  processes,  and  procedures  are  needed 
to  enable  effective  centralized  test  control. 
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These  measures  determined  the  degree  to  which  test  exercise/control  features  could  be  improved 
through  experimentation,  and  the  degree  to  which  protocols,  processes,  and  procedures  were 
needed  to  enable  effective  centralized  test  control  in  a  distributed  environment.  JADS  personnel 
analyzed  test  logs,  test  team  discussions,  and  after-action  reports  to  determine  the  effectiveness 
of  the  test  control  procedures  and  if  the  test  exercise/control  features  had  been  improved. 

The  test  exercise/control  procedures  used  during  the  Phase  4  test  were  developed  and  refined 
during  the  previous  test  phases.  These  procedures  included  standardized  checklists  addressing 
the  start-up  and  shutdown  of  the  ETE  Test  network  and  the  conduct  of  each  test  trial.  The 
network  checklist  included  procedures  to  verify  network  connectivity,  data  storage  space,  and 
time  server  accuracy,  and  to  test  network  transmission  prior  to  test  start.  The  test  conduct 
checklist  included  detailed  start-up  procedures  for  each  test  node.  These  checklists  are  provided 
in  Appendix  A  to  this  report. 

Communication  procedures  were  also  developed  during  the  previous  test  phases.  Based  on  the 
experience  gained  during  these  tests,  the  Phase  4  testers  were  able  to  use  the  conference  phone 
lines  to  effectively  resolve  system  failures  and  to  facilitate  useful  discussions  of  the  after-action 
reports.  The  only  exception  to  this  occurred  during  the  first  live  flight  on  19  March.  Numerous 
communication  problems  were  experienced  during  the  flight  because  of  failed  communication 
links  and  inadequate  test  start-up/communication  procedures  between  the  TCAC  and  the  aircraft. 
These  problems  were  resolved  once  the  aircraft  arrived  at  Fort  Hood  and  established  contact  with 
the  LGSM.  Following  the  19  March  flight,  specific  procedures  for  contacting  the  aircraft  and 
TCAC  were  established  along  with  revised  procedures  for  start-up  during  live  flights. 

Effective  centralized  test  control  was  exercised  through  the  test  controller  located  at  the  TCAC. 
The  test  controller  was  responsible  for  starting  and  stopping  each  trial,  monitoring  the  status  of 
all  nodes  and  network  links,  and  declaring  holds  and  restarts  when  necessary.  The  test  controller 
was  able  to  continuously  monitor  the  status  of  the  trials  through  a  combination  of  network 
monitors  in  the  TCAC  and  voice  communications  with  key  personnel  at  each  site.  These 
procedures  proved  effective  for  timely  detection  and  resolution  of  system/network  failures. 

Test  exercise/control  in  a  distributed  computing  environment  offers  many  challenges  not 
experienced  in  conventional  testing.  The  designers  of  a  test  network  should  carefully  plan  for 
test  exercise/control,  noting  that  good  communication  is  required  among  all  network  nodes. 
Otherwise,  poor  test  control  can  result  in  inaccurate  and  untimely  information  being  disseminated 
throughout  the  test  architecture  and  can  waste  valuable  test  time  and  test  assets.  Such  test 
exercise/control  features  should  also  be  examined  following  testing  activities.  Lessons  learned 
from  experimentation  in  an  ADS  environment  should  be  applied  to  test  exercise/control  in  order 
to  improve  how  future  tests  in  the  ADS  environment  are  conducted. 
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5.0  Lessons  Learned 


5.1  Technical  Lessons  Learned 

5.1.1  Simulations 

When  used  in  ADS  testing,  simulations  should  be  carefully  planned  and  developed.  The  ADS 
tester  must  also  be  in  close  communication  with  each  organization  modifying  simulations  crucial 
to  the  test.  The  ETE  Test  Phase  4  succeeded  because  of  the  relatively  high  degree  of  reliability 
of  the  simulations  used  in  the  test.  If  simulation  execution  problems  had  occurred,  the  ETE  Test 
team  had  arranged  for  a  quick  response  by  the  personnel  needed  to  fix  those  difficulties. 

Simulation  software  must  be  kept  under  strict  configuration  control.  If  an  error  is  found  in  the 
software,  software  configuration  management  allows  the  error  to  be  identified  with  a  particular 
version  of  the  software  and  ensures  that  the  needed  software  fix  will  be  included  in  all  later 
versions. 

5.1.2  Interfaces 

Distributed  testing  often  requires  linkage  among  dissimilar  facilities,  network  equipment,  and 
simulations.  However,  careful  planning  can  significantly  reduce  the  potential  for  difficulties 
arising  from  network  interface  problems.  As  part  of  its  planning,  the  ETE  Test  team  bought 
standard  network  equipment  for  all  the  sites.  Thus,  the  configuration  of  the  ETE  Test 
environment  did  not  pose  any  problems  during  the  Phase  4  test. 

5.1.3  Networks 

Time  sources  must  be  synchronized.  Time  must  be  synchronized  from  the  same  time  source, 
then  validated  at  each  network  node  to  ensure  that  accurate,  synchronized  time  is  recorded  at 
each  node.  The  time  server  used  in  the  ETE  Test  worked  very  well,  ensuring  that  all  loggers 
were  set  at  the  same  time  and  keeping  time  differences  between  loggers  to  a  minimum.  Having 
the  same  time  at  all  loggers  helped  speed  up  the  analysis  and  allowed  for  the  use  of  automated 
analysis  tools. 

The  use  of  the  satellite  link  during  Phase  4  resulted  in  the  existence  of  two  networks.  The  WAN 
that  was  time  synched  based  upon  a  GPS  signal,  and  a  LAN  internal  to  the  aircraft  that  was  also 
time  synched  by  a  GPS  signal.  There  was  no  way  to  compare  the  two  times,  so  the  assumption 
was  maHp.  that  the  two  time  domains  were  within  the  test’s  established  tolerance  of  one 
millisecond.  This  is  a  reasonable  assumption  for  a  test  of  a  man-in-the-loop  C4ISR  system. 

5.1.4  Instrumentation 

Special  equipment  was  necessary  for  ADS  network  check-out  and  verification.  Special  test 
equipment  and  networking  tools  will  rapidly  isolate  the  specific  cause  of  network  and  ADS/DIS 
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problems.  Without  the  special  equipment,  troubleshooting  would  have  been  accomplished  by 
trial  and  error  increasing  time,  cost,  and  personnel.  In  addition,  the  key  Network  and 
Engineering  personnel  should  be  trained  in  the  use  of  the  special  test  equipment  and  networking 
tools. 

The  flow  and  transfer  of  PDUs  were  critical  to  the  ETE  Test.  PDU  traffic  was  continuously 
monitored  at  all  nodes  to  ensure  that  PDUs  were  constantly  flowing.  Since  the  ETE  Test 
manning  levels  were  insufficient  to  enable  watching  these  monitors  at  all  time,  problems  that 
interrupted  PDU  flow  were  not  always  inunediately  noticed.  Some  type  of  audio  warning  device 
might  have  reduced  test  time  loss  from  a  few  minutes  to  seconds. 

5.2  Infrastructure  and  Process  Lessons  Learned 

5.2.1  Procedures 

5.2. 1.1  Planning 

—  The  requirements  for  an  ADS  test  must  be  clearly  defined  early  in  the  test  planning  phase. 
Detailed  planning  and  coordination  are  required  to  ensure  a  common  understanding  of  all 
requirements,  procedures,  and  test  objectives  since  individual  facilities  are  generally 
unfamiliar  with  conducting  coordinated,  distributed  T&E  tests. 

—  When  planning  for  the  integration  of  a  major  activity  into  the  ADS  environment,  provide 
plenty  of  time  for  risk  reduction  testing.  In  addition,  allow  time  between  major  test  events  to 
fix  problems  should  they  occur.  For  example,  downtime  was  scheduled  following  the  25 
March  live  flight  to  allow  resolution  of  any  problems  that  occurred  during  the  flight  and 
before  the  next  test  event.  Only  minor  problems  were  experienced,  and  there  was  sufficient 
time  to  resolve  these  difficulties  before  the  31  March  live  flight. 

—  Be  open  to  new  ideas,  as  some  of  the  old  ideas  from  the  early  stages  of  an  ADS  test  program 
may  become  very  expensive  to  bring  to  fmition.  The  ETE  Test  Phase  4  was  originally  slated 
to  have  nine  scenarios.  As  the  requirements  for  each  scenario  increased,  development  costs 
also  grew.  These  added  costs  eventually  led  to  the  deletion  of  the  last  four  scenarios. 

—  Minimize  the  impact  of  hardware  problems.  When  using  a  complicated  ADS  network  with  a 
vast  array  of  equipment,  hardware  problems  will  occur.  Plan  in  such  a  way  that  unexpected 
hardware  problems  do  not  completely  disrapt  the  test.  During  the  ETE  Test  Phase  4,  steps 
were  taken  to  ensure  that  hardware  problems  did  not  dismpt  the  test  for  long  periods  of  time. 
For  example,  data  saves  were  accomplished  frequently.  In  addition,  the  network  was 
constantly  monitored  to  ensure  that  hardware  problems  were  fixed  as  soon  as  possible. 

5.2. 1.2  Development 

—  Use  a  stepping  stone  approach  to  testing  where  each  successive  ADS  test  builds  on  the 
success  of  earlier  tests.  This  “test,  analyze,  fix,  test”  approach,  in  concert  with  structured. 
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independent  testing  of  the  network,  will  greatly  improve  the  chances  for  successful  ADS 
testing. 

-  Risk  reduction  testing  prior  to  actual  test  execution  will  help  test  team  personnel  identify  and 
resolve  potential  ADS  system  problems. 

5.2. 1.3  Execution 

-  Briefings  are  needed  before  each  ADS  test.  These  briefings  should  include  such  information 
as  the  test  objectives,  telephone  numbers  to  use  for  test  control,  the  test  configuration  of  each 
facility,  instrumentation  and  data  collection  requirements,  go/no  go  criteria,  contingency  and 
backup  plans,  and  test  conduct.  A  briefing  checklist  should  be  developed  and  used.  These 
procedures  should  be  established  and  understood  by  all  ADS  test  participants.  In  addition, 
these  procedures  should  be  reviewed  for  accuracy  when  requirements  or  nodes  are  added  to 
the  test. 

-  Test  control  procedures  should  be  well  rehearsed.  When  many  people  are  communicating  on 
one  phone  line,  a  response  order  should  be  established  and  strictly  followed  to  save  valuable 
test  time. 

-  Communication  and  coordination  among  ADS  test  team  members  are  vital  for  the  success  of 
an  ADS  test  especially  when  changes  or  additions  are  made  to  the  test  environment. 

-  Do  not  allow  test  personnel  to  use  test  equipment  for  purposes  other  than  the  test  being 
conducted.  Use  of  one  of  the  loggers  to  compile  test  analysis  code  during  the  conduct  of  a 
trial  caused  PDUs  to  not  be  recorded  and  resulted  in  an  incomplete  log  file. 

5. 2. 1.4  Evaluation 

-  Effective  data  management  is  needed.  Linked  facilities  can  generate  a,  large  volume  of  data 
at  distributed  locations.  Without  careful  planning,  key  data  may  not  be  collected  and/or 
transmitted  to  the  analysis  center,  and  data  collected  at  the  network  nodes  may  not  be  in  a 
useful  form  for  centralized  analysis.  Before  ADS  testing,  a  comprehensive  data  management 
plan  must  clearly  identify  the  data  to  be  collected  at  each  network  node,  on-site  processing  of 
the  data,  and  the  data  to  be  transmitted  to  the  analysis  center. 

-  Pretest  rehearsals  are  also  useful  for  improving  evaluation  procedures.  The  ETE  Test  team 
improved  its  data  collection  and  analysis  processes  as  a  result  of  experiences  from  the 
functionality  and  integration  and  risk  reduction  tests. 

5.2. 1.5  Command  and  Control 

-  Have  test  controllers  who  are  extremely  familiar  with  the  test  and  network  configuration. 
The  Phase  4  test  succeeded  partly  because  it  had  an  experienced  test  controller  with  the 
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necessary  tools  to  evaluate  problems  and  the  authority  to  make  meaningful  decisions 
regarding  test  problems. 

-  Have  a  centralized  test  control  center.  The  JADS  TCAC  is  configured  to  allow  for 
convenient,  instant  communications  with  all  the  nodes.  It  acted  as  the  central  point  of  contact 
between  the  nodes  and  for  all  problems.  The  test  controller  kept  track  of  test  progress  and 
documented  any  problems  that  occurred. 

-  Establish  control  over  resources.  Linking  various  facilities  using  ADS  can  require  significant 
development  of  facility  interface  hardware  and  software. 

-  Distributed  tests  require  personnel  distribution.  When  many  distributed  nodes  are  required 
for  the  successful  completion  of  a  test,  personnel  will  need  to  be  located  at  these  nodes.  The 
complexity  and  input  an  individual  node  contributes  should  guide  the  assignment  of 
personnel.  The  ETE  Test  Phase  4  required  several  people  at  the  Fort  Hood,  Northrop 
Grumman  and  TCAC  nodes;  only  one  person  was  needed  at  the  White  Sands  node.  The  Fort 
Sill  node  used  only  resident  personnel. 

5.2.2  Policy 

-  Network  management  and  troubleshooting  must  be  disciplined  and  organized  with  a 
thorough  understanding  and  strong  configuration  control  of  the  ADS  network. 

5.2.3  Personnel 

-  Personnel  involved  in  a  distributed  test  should  understand  the  “big  picture.”  When  people 
are  geographically  separated,  it  becomes  easy  for  them  to  focus  on  their  own  individual 
portion  of  the  test.  When  problems  arise,  personnel  who  understand  the  entire  test  and  the 
overall  network  will  find  solutions  much  faster. 


68 


6.0  Conclusions/Recommendations 


6.1  Utility 

6.1.1  Utility  Conclusions 

6.1. 1.1.  Enhanced  Testing.  An  ADS-enhanced  live  test  environment  can  enhance  the  testing  of 
C4ISR  systems. 

-  The  Phase  4  test  showed  that  ADS  technology  can  realistically  represent  a  C4ISR  system 
enabling  the  system’s  users  to  collect  valid  and  useful  operational  information. 

-  Compared  to  conventional  methods,  an  ADS-enhanced  live  test  environment  can  realistically 
test  C4ISR  systems: 

-  with  larger  numbers  of  ground-based  entities  at  a  much  lower  cost. 

-  with  more  control  over  the  specific  aspects  of  the  scenarios  being  tested.  Using  ADS,  the 
test  director  is  not  at  the  mercy  of  a  training  exercise  over  which  he/she  has  no  direction. 
Rather,  the  test  director  can  control  the  simulated  entities. 

-  for  longer  periods  of  time  enabling  increased  data  collection  and  the  ability  to  analyze  and 
improve  the  data  gathering  process. 

-  under  realistic  but  unsafe  conditions,  such  as  convoy  vehicles  driving  across  rugged 
terrain  under  wartime  conditions. 

-  By  allowing  the  simulation  of  large  battlespaces  with  large  numbers  of  entities,  ADS 
technology  provides  testers  with  greatly  expanded  capabilities  for  test  concept  and  design. 

-  Testers  can  use  ADS  to  save  time,  resources  and  test  personnel  man-hours  by  linking  several 
pieces  of  equipment  and/or  facilities  together  for  simultaneous  testing  instead  of  conducting 
individual  tests  at  different  locations. 

-  ADS  technology  provides  access  to  elements  participating  in  the  test  from  their  normal  work 
locations,  greatly  reducing  the  additional  operational  tempo  required  to  support  both  testing 
and  test  integration,  training  and  rehearsal. 

6.1. 1.2  Improved  Opportunities  for  Training.  If  an  ADS  environment  is  developed  for  testing, 
the  same  environment  can  easily  be  modified  and  transitioned  to  the  training  world.  ADS 
technology  can  then  improve  opportunities  for  training  with  a  C4ISR  system. 


69 


-  Conventional  training  can  be  limited  by  the  availability  of  those  assets  making  up  the  C4ISR 
system’s  operational  environment.  ADS  technology,  by  simulating  those  key  assets,  can 
provide  longer  periods  of  time  for  realistic  operation  of  the  C4ISR  system. 

-  C4ISR  system  operators  can  take  advantage  of  the  additional  training  time  provided  by  ADS 
technology  to  confirm  current  tactics  and  to  test  “what  if’  scenarios  and  new  tactics. 

-  A  C4ISR  simulation  incorporated  into  an  ADS  environment  can  enhance  operator  training  by 
providing  useful  information  on  the  battlefield  surveillance  and  the  targeting  process.  It  can 
also  help  the  battle  manager  by  providing  a  high-level  picture  of  the  entire  battlefield  and 
aiding  in  the  effective  allocation  of  battle  resources. 

-  An  ADS  environment  can  provide  testers  and  C4ISR  system  operators  with  the  opportunity 
to  check  the  interoperability  and  compatibility  of  the  equipment. 

-  ADS  simulations  can  help  C4ISR  system  operators  familiarize  themselves  with  the  maneuver 

*  tactics  of  foreign  armed  forces  —  valuable  experience  for  possible  future  deployments. 

-  C4ISR  system  operators  can  train  in  an  operational  environment  with  multiple  assets  using 
the  capabilities  of  ADS  technology  to  tie  several  pieces  of  equipment  and/or  facilities 
together  and  to  simulate  large  numbers  of  fielded  vehicles  and  associated  personnel.  In 
contrast,  conventional  training  would  provide  far  fewer  assets  in  the  operational  environment, 
if  any  at  all. 

6.1.2  Utility  Recommendations 

-  Large  exercises  could  use  the  ETE  Test  environment  to  virtually  augment  the  battlefield  with 
simulated  targets.  During  Phase  4,  this  capability  was  demonstrated  with  the  integration  of  a 
live  E-8C  Joint  STARS  aircraft  into  the  ETE  Test  ADS  environment. 

-  An  ADS-enhanced  live  test  environment,  like  the  ETE  Test  environment,  is  flexible  enough 
to  allow  for  further  expansion  and  increased  opportunities  for  testing  C4ISR  systems.  The 
Janus  battlespace  can  be  expanded  as  required.  Increasing  the  number  of  LGSMs  or  common 
ground  stations  (CGSs)  would  create  more  realistic  targeting  capabilities.  By  adding  other 
assets  to  the  environment,  such  as  an  unmanned  aerial  vehicle  (UAV)  or  a  tactical  aircraft 
simulator,  the  robustness  of  the  environment  could  be  significantly  enhanced. 

6.2  Technical 

6.2.1  Technical  Conclusions 

-  The  ETE  Test  Phase  4  WAN  required  only  a  small  portion  of  the  available  bandwidth.  This 
indicates  that  a  much  higher  packet  rate  could  be  applied  to  ADS  testing  without  causing 
bandwidth  problems.  The  satellite  link,  however,  limited  the  packet  rate  available  for  live 
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testing.  This  required  modifications  to  the  DIS  standard  (no  heartbeat  and  creation  of  VDPs) 
and  may  not  be  acceptable  in  all  instances.  The  weakest  link,  or  in  this  case  the  smallest, 
must  be  identified  and  the  test  designed  to  accommodate  it. 

-  The  ETE  Test  network  was  highly  reliable  and  stable  during  the  Phase  4  test. 

-  The  ETE  Test  team’s  extensive  risk  reduction  testing  played  an  important  role  in  eliminating 
the  presence  of  major  ADS  system  failures  during  Phase  4  testing.  Many  ADS  problems 
were  identified  and  resolved  during  this  risk  reduction  period,  thus  minimizing  their  potential 
for  affecting  the  actual  testing.  As  a  result,  the  ADS  problems  noted  in  Table  9  had  only  a 
minor  impact  on  Phase  4  testing  and  caused  only  brief  delays  or  reductions  in  capability. 

6.2.2  Technical  Recommendations 

-  With  careful  planning  and  resource  management,  testers  can  address  the  issues  associated 
with  integrating  simulations  into  an  ADS  test  environment.  They  can  also  identify  the 
assumptions  and  limitations  associated  with  those  simulations. 

-  Budget,  schedule,  and  provide  the  manpower  necessary  to  develop  the  simulations. 
Simulation  development  is  typically  labor  intensive  and  thus  costly. 

-  Determine  the  level  of  simulation  detail  needed  for  the  ADS  test.  Development  costs  are 
directly  related  to  the  level  of  simulation  detail. 

-  Identify  and  provide  training  for  the  users  of  the  simulations. 

6.3  Infrastructure 

6.3.1  Infrastructure  Conclusions 

-  ADS  can  reduce  the  number  of  troops  and  associated  equipment  involved  in  tests  because  of 
its  simulation  of  fielded  forces.  However,  the  ADS  infrastructure  requires  technical 
personnel  to  set  up  and  execute  the  tests  and  to  analyze  the  test  results. 

-  Highly  structured  test  control  is  a  key  ingredient  for  ADS  test  success.  This  test  control 
should  include  formalized  procedures  with  an  emphasis  on  checklists. 

-  An  ADS  test  can’t  always  count  on  having  the  personnel  requirements  for  a  distant  node 
supplied  by  an  organization  local  to  the  node.  Even  if  an  ADS  test  is  able  to  employ  these 
people,  it  may  then  lose  them  to  other  activities  deemed  more  important  by  the  local 
organization. 

-  An  ADS-enhanced  live  test  environment  necessitates  sophisticated  instrumentation  with 
rigorous  processing  speed,  data  storage,  and  data  integration  capabilities.  This 
instrumentation  can  be  costly  and  can  require  trained  personnel  for  its  successful  operation. 
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—  Distributed  testing  typically  means  distributed  personnel  and  distributed  equipment. 
Distributed  personnel  lead  to  high  travel  costs.  Equipment  located  at  distant  network  nodes 
will  still  require  maintenance  either  through  contracts  or  trips  by  a  network  engineering  team. 

—  ADS  analysts  must  have  a  well-planned  and  organized  approach  to  managing  the  large 
amounts  of  data  produced  from  ADS  testing. 

6.3.2  Infrastructure  Recommendations 

-  Make  every  effort  to  simplify  the  infrastructure.  Time  spent  in  the  planning  stages  of  an  ADS 
test,  with  an  emphasis  on  reducing  the  complexity  of  the  test  network,  is  time  well  spent. 
Use  proven  hardware  and  keep  it  the  same  wherever  possible. 

-  Keep  in  mind  the  disadvantages,  as  well  as  benefits,  of  the  networked  nature  of  an  ADS 
environment.  The  ADS  tester  will  almost  always  be  dependent  upon  a  telecommunications 
provider. 


72 


Appendix  A  -  JADS  Test  Procedures 


Al.O  Test  Procedures 

Various  types  of  checklists  were  used  during  the  execution  of  the  Phase  4  test.  The  Test  Control 
and  Analysis  Center  (TCAC)  test  controller  checklist  can  be  found  in  Section  Al.l,  TCAC  Test 
Procedures.  This  checklist  was  used  to  ensure  network  and  logger  functionality  and  to  provide 
overall  test  control  procedures.  Each  node  (White  Sands  Missile  Range  [WSMR],  Northrop 
Grumman,  and  Fort  Sill)  incorporated  the  logger  functions  from  the  TCAC  checklist  into  their 
own  checklist. 

Other  checklists  were  used  to  direct  the  operation  of  various  pieces  of  test  equipment.  An 
example  is  included  in  Section  A1.2,  TCAC  Plan  View  Display  (PVD)  Procedures. 

Section  A1.3,  WSMR  Procedures,  is  representative  of  the  site-specific  checklists.  WSMR, 
Northrop  Grumman  and  Fort  Sill  all  developed  procedures  for  operation  of  the  End-to-End 
(ETE)  Test  environment  equipment.  Only  Fort  Hood,  the  only  site  without  a  logger,  failed  to 
develop  written  procedures.  Their  procedures  were  primarily  accomplished  by  the  resident 
specialists  who  have  their  own  procedures. 

Al.l  TCAC  Test  Procedures 


The  following  are  the  written  test  procedures  used  in  the  TCAC  during  Phase  4  testing. 

72  HOURS  PRIOR  TO  TEST 

Network  Coordinator:  _ _ _ _ 

Date: _ Test  Time: _ to _ 

1 _ Check  supplies. 


2  Turn  on  equipment. 

^ _  a.  Turn  on  3  Barcos  (Spectrum  [Sun5]  on  1,  Janus  [hp735]  on  2,  and  NetVis  [indigo2]  on 

3). 

_  b.  Log  in  as  “root”  to  indigo2  in  the  TCAC,  and  indy4  in  communications  room  1. 

1)  From  the  toolchest,  select  Toolbox,  JADS  Toolbox,  Monitor,  PDU  Monitor, 

_  PDU  Statistics,  Show  Stats  to  display  protocol  data  units  (PDUs). 

_  2)  From  the  toolchest,  select  NetVis,  NetGraph-ETE  to  display  network  traffic. 

3)  From  the  toolchest,  select  NetTests,  Status  Check  ETE  to  start  and  display 
network  connectivity  tests,  (uts  in  conun  rm  1  pings  wsmr,  ftsill,  and  fthoodafatads. 

_  indigo2,  pings,  grumman,  indy3,  and  sparcS  at  Ft  Hood). 

c.  In  the  TCAC,  run  Spectram  on  the  Sun20  (server)  and  SunS  (graph)  to  display  Zulu 
_  time  and  router  status. 

d.  Create  an  empty  file  “touch  /  scripts  / .  go”  in  grumman,  indigo2,  and  indy4. 
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Clear  router  interfaces.  To  clear  the  grumman_router,  jads_router,  and  fthood_router  from 
indigo2;  and  fthood_router,  ftsiIl_router,  and  wsmr_router  from  indy4,  run: 
“/scripts/clear_router  ete.” 

4 _ Not  used. 


5  Time  accuracy.  Verify  that  each  site  has  network  time  protocol  (NTP)  running 

= -  a.  From  uts,  run  “/scripts/check_tinie”  and  verify  that  the  offsets  for  ftsill  and 

wsmr  are  less  than  1  millisecond  (ms). 

-  b.  From  indigo2,  rlogin  to  indyl,  run  “/scripts/check_time . ”  Verify  offsets  for 

grumman,  indy3,  and  sparcS  are  less  than  1  ms. 

6  Available  disk  space.  Verify  that  each  logger  has  at  least  600  megabytes  (MB)  of  unused  disk 
space  available  on  the  /disk2  partition. 

- - a.  From  uts,  rlogin  to  ftsill  and  wsmr,  in  turn,  and 

from  indigo2,  rlogin  to  grumman,  and  indy3,  in  turn. 

-  b.  Run  “df  -k”  on  each  machine  (including  uts)  to  display  the  available  disk  space. 

Verify  that  each  has  at  least  600  MB  available. 

7  Port  settings.  Verify  that  each  logger  is  set  to  port  3000  and  the  exercise  identification  (ID)  is  0. 

= - a.  From  uts,  rlogin  to  ftsill  and  wsmr,  in  turn,  and 

from  indigo2,  rlogin  to  grumman,  and  indy3,  in  turn. 

- b.  Run  “more  /scripts/dt_logger”  to  view  the  file.  Look  for  the  entry: 

“ /usr/ local /bin/ jads_logger  3000  0  /disk2/logfiles 
/$testdate"_test"$testnum"_"$runnuia"_"$site".log"  ”  entry  in 
two  places. 

§ - Voice  conference  net.  Verify  the  net  is  functional  by  dialing  in  from  two  different  phones  in  the 

TCAC  at  the  same  time  to  establish  the  net. 

2 _  Not  used. 


1  Data  collection  test  a: 

0 -  a.  From  uts,  rlogin  to  ftsill  and  wsmr,  simultaneously,  and 

from  indigo2,  rlogin  to  grumman,  and  indy3,  simultaneously. 

-  b.  Start  the  ftsill,  grumman,  indy3,and  uts  loggers  using  test  number  “000”  and  run 

number  “a”  (i.e.  -  ‘7scripts/dt_logger  000  a”). 

-  c.  Run  the  “/scripts/run_player  3000  /disk2/logfiles/ne_test  ;log” 

file  on  the  wsmr  machine. 

- d.  Determine  when  run  is  complete.  Stop  all  loggers  (“Ctrl-C”). 

-  e.  Check  digital  communications  terminal  (DCT)  results.  Verify  reception  of  2281  PDUs 

on  grumman,  indy3,  and  uts  (or  indy4)  loggers.  (No  PDUs  at  ftsill). 

1  Data  collection  test  b: 

1 -  a.  From  uts,  rlogin  to  ftsill  and  wsmr,  simultaneously,  and 

from  indigo2,  rlogin  to  grumman,  and  indy3,  simultaneously. 
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_  b.  Start  the  grumman,  indyS,  uts  and  wsmr  loggers  using  test  number  “0 00”  and  run 

number  “a”  (i.e.  -  “/scripts/dt_logger  000  a”). 

_  c.  Run  the  “/scripts/run_player  3000  /disk2 /logf  lies /ne_test  .log” 

file  on  the  ftsill  machine. 

_ d.  Determine  when  run  is  complete.  Stop  all  loggers  (“Ctrl-C”). 

_  e.  Check  DCT  results.  Verify  reception  of  2281  PDUs  on  grumman,  indy3,  and  uts 

loggers. 

1  _ Report  the  results  of  the  network  checks  to  the  test  controller.  Supervise  repairs  as  necessary  to 

2  prepare  equipment  for  the  test  sequence. 


PRETEST  (DAY  OF  TEST) 

Network  Coordinator: _ _ _ 

Date: _ Test  Time: _ to _ 

] _ Check  supplies.  Provide  checklists,  blank  log  sheets,  file  name  lists,  pens,  pencils,  scratch 

paper,  and  4  millimeter  (mm)  tape  cartridges  for  the  test. 

2  Turn  on  equipment. 

. _  a.  Turn  on  3  Barcos  (Spectrum  [Sun5]  on  1,  Janus  [hp735]  on  2,  and  NetVis  [indigo2]  on 

3). 

_  b.  Log  in  as  “root”  to  indigo2  in  the  TCAC,  and  indy4  in  conununications  room  1 . 

1)  From  the  toolchest,  select  Toolbox,  JADS  Toolbox,  Monitor,  PDU  Monitor, 

_  PDU  S^tatistics,  Show  Stats  to  display  PDUs. 

_  2)  From  the  toolchest,  select  NetVis,  NetGraph-ETE  to  display  network  traffic. 

3)  From  the  toolchest,  select  NetTests,  Status  Check  ETE  to  start  and  display 
network  connectivity  tests,  (uts  in  Comm  Rm  1  pings  wsmr,  ftsill,  and  fthoodafatads. 

_  indigo2,  pings,  grumman,  indyS,  and  sparc5  at  Ft  Hood). 

c.  In  the  TCAC,  run  Spectrum  on  the  Sun20  (server)  and  SunS  (graph)  to  display  Zulu 
time  and  router  status. 

3  Clear  router  interfaces.  To  clear  the  grumman_router,  jads_router,  and  fthood_router  from 
indigo2;  and  fthood_router,  ftsill_router,  and  wsmr_router  from  indy4,  run: 

' _ “/scripts/clear_router  ete.” 

4  _ Not  used. 

5  Time  accuracy.  Verify  that  each  site  has  NTP  running. 

, _  a.  From  uts,  run  “/scripts/check_time”  and  verify  that  the  offsets  for  ftsill  and 

wsmr  are  less  than  1  ms. 

_  b.  Fromindigo2,  rlogin  to  indyl,  run  “/scripts/check_time.”  Verify  offsets  for 

grumman,  indy3,  and  sparcS  are  less  than  1  ms. 
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6 _ Available  disk  space.  Performed  at  each  logger  by  the  logger  operator. 


7 _ Port  settings.  Performed  at  each  logger  by  the  logger  operator. 


8 _ Join  the  voice  conference  net.  Both  the  test  controller  and  the  network  coordinator  (NC)  dial 

61143  in  the  TCAC  to  establish  the  conference  net. 

2 - Time  synchronization,  ftsill,  grumman,  indy3,  and  wsmr  operators  check  global  positioning 

system  (GPS)  time  reception  by  typing  “date”  and  press  Enter  on  the  NC’s  mark.  Report  time 
to  NC. 

(NOTE:  indyl  is  time  server  for  classified,  uts  is  time  server  for  unclassified.) 

1  Data  collection  test  a: 

Q -  a.  Cue  ftsill,  grumman,  indy3,and  uts  operators  to  start  loggers  using  test  number  “0 00” 

and  ran  number  “a”  (i.e.- “/script s/dt_logger  000  a”). 

_  b.  Cue  wsmr  operator  to  ran  “/scripts /ruii_player  3000 

/disk2/logf iles/ne_test . log”  file. 

-  c.  Determine  when  ran  is  complete.  Cue  all  operators  to  stop  loggers  (“Ctrl-C”). 

- d.  Check  DCT  results  -  Have  grumman,  indy3,  and  uts  operators  verify  reception  of  2281 

PDUs.  (NoPDUsatftsiU). 

1  Data  collection  test  b: 

1 -  a.  Cue  grumman,  indy3,  uts,  and  wsmr  operators  to  start  loggers  using  test  number 

“000”  and  run  number  “b”  (i.e.  -  “/scripts/dt_logger  000  b”). 

-  b.  Cue  ftsill  operator  to  ran  “/scripts/run_player  3000 

/disk2/log£iles/ne_test .log”  file. 

-  c.  Determine  when  run  is  complete.  Cue  all  operators  to  stop  loggers  (“Ctrl-C”). 

- d.  Check  DCT  results  -  Have  grumman,  indy3,  uts,  and  wsmr  operators  verify  reception  of 

2281  PDUs.  (No  PDUs  at  ftsill). 

1  _ Report  the  results  of  the  network  checks  (items  9-11)  to  the  test  controller. 

2 


The  pretest  phase  is  now  complete.  Proceed  to  the  test  ran  phase. 

NOTE:  Sometimes  the  logger  process  does  not  terminate  on  the  grumman  logger.  In  that  case,  run 
/scripts/f  ind_logger  on  the  grumman  logger  to  kill  the  process  and  delete  the  old  log  file 
before  restarting  the  logger  with  the  same  filename. 
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TEST  RUN 


Network  Coordinator: _ 

Date: _  Lab  Time: _ to _ 

1  _ _  Start  loggers.  Obtain  the  test  and  mn  numbers  from  the  test  controller  and  record  on  the  log 

sheet.  Operators  are  cued  by  the  test  controller  when  to  start  loggers.  Record  start  time  on  the 
log  sheet. 

_  a.  Early  in  the  test  ran,  verify  with  operators  that  all  loggers  are  receiving  PDUs  (number 

is  increasing. 

_  b.  Periodically  check  with  operators  that  all  loggers  continue  to  receive  PDUs  (number  is 

increasing). 

_  c.  Periodically  check  “bat”  phone  operation  if  not  used  regularly. 

_  d.  Every  1/2  hour,  ran  a  time  accuracy  check.  From  uts,  ran 

“/ script s/check_time”  to  check  ftsill  and  wsmr.  From  indigo2,  rlogin  to  indyl 
and  ran  “/scripts/check_time”  to  check  indyl,  grumman,  and  tcacindy.  Time  offsets 

_  should  be  <1  ms. 

e.  Keep  written  event  log. 

2  Stop  loggers.  Loggers  stop  recording  data  when  directed  by  the  test  controller  (“Ctrl-C”). 

.  _ a.  Record  the  stop  time  and  the  total  number  of  PDUs  from  each  logger  on  log  sheet. 

_  b.  Confirm  that  the  required  data  have  been  logged.  From  uts,  rlogin  to  ftsill  and  wsmr 

and  ran  “Is  -1  /disk2/logfiles.”  Fromindigol,  rlogin  to  indyS,  and 
grumman  and  ran  “Is  -1  /disk2/log£iles.”  Check  the  file  sizes;  the  filename  is 
“mmddyyJest#-run^_loggermme. log” 

3  _  Subsequent  runs.  When  additional  runs  are  required,  repeat  steps  1  and  2  for  each  ran. 


The  test  ran  phase  is  now  complete.  Proceed  to  the  post-test  phase. 


POST  TEST  (DAY  OF  TEST). 

Network  Coordinator: _ 

Date: _  Test  Time: _ to _ 

1  Remote  file  capture.  Consolidate,  compress,  and  copy  the  test  ran  logger  files  from  each  remote 

site. 

_  a.  For  classified  data,  rlogin  to  each  logger  (grumman  and  indy 3),  in  turn,  from  tcacindy 

in  the  TCAC,  or 

For  unclassified  data,  rlogin  to  each  logger  (ftsill,  wsmr,  and  uts),  in  turn,  from  uts. 

_  b.  If  only  1  file  for  the  day  exists  in  the  logger  at  a  site,  skip  to  step  c.  If  more  than  1  log  file 

for  the  day  exists  at  a  site,  consolidate  them  by  using  the  command 
“tar  cvf  jnmddyy_sitenaine.log. tar  jnmddyy* . log ” 
where  *  is  the  wildcard  character  that  includes  all  the  files  for  that  day  for  that  site  name. 
(e.g.,-“tar  cvf  040798_wsinr.log. tar  040798*  .log”  ). 
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-  c.  Compress  the  single  log  file  (e.g.,  -  “compress  040798_wsmr.log”)  or  the  tar  file 

from  step  b  (“compress  040798_wsmr .  tar”). 

-  d.  On  tcacindy,  run  “/scripts/rcp_etef  ile”  to  copy  the  tar'd  and  compressed 

classified  files  (^'mmddyy_sitename.\ogJL”)  from  both  grumman  and  indy3  loggers  to 
tcacindyr/diskl/ete/mmddyy/. 

- ^  e.  On  uts,  run  “/scripts/rcp_etef  ile”  to  copy  the  tar'd  and  compressed 

unclassified  files  {“mmddyy_sitename.log.z”)  from  ftsill,  uts,  and  wsmr  loggers  to 

-  uts:/disk2/ete/inmddyy/. 

-  f.  Copy  the  unclassified  files  from  step  e  to  4mm  tape  and  activate  the  write  protect  feature. 

g.  Move  the  tape  to  tcacindy  and  copy  the  unclassified  files  from  the  tape  to 
Idiskl/tte/mmddyyl  (i.e.,  from  the  ete  directory,  run  tar  xv  to  extract  the  files  from  the 
tape  to  the  hard  drive).  Make  sure  the  write  protect  feature  is  ON. 

2  Backun  tapes. 

• -  a.  Create  a  backup  tape  of  the  files  in  tcacindy:/disk2/ete//ninddyy/  using  either  the  “  tar 

cv  mmddyy”  command  while  in  the  ete  directory  (or  the  tape  tool  on  the  tcacindy 
desktop). 

-  b.  Verify  the  backup  using  either  the  “tar  tv”  command  or  the  tape  tool. 

- c.  Remove  the  tape  from  the  drive  and  label  it. 

- d.  Repeat  a,  b,  and  c  to  create  a  duplicate  tape. 

e.  Deliver  both  tapes  to  the  ETE  Test  team  representative. 

3  -  Delete  the  data  collection  test  and  the  backed-up  log  files  from  /disk2/logfiIes/  on  all  loggers. 

4  - On  the  last  day  of  testing,  delete  the  file  “/scripts  /  .go”  in  gramman,  indigo2,  and  indy4. 

5  - Logoff  from  logger.  Turn  Off  the  monitor,  but  leave  the  central  processing  unit  (CPU)  On! ! ! 

6  _ Participate  in  mission  debrief,  if  applicable. 


78 


A1.2  TCAC  Plan  View  Display  (PVD)  Procedures 


The  following  procedures  were  used  to  initiate  test  monitoring  with  the  Janus  Plan  View  Display 
program.  This  is  representative  of  the  specific  checklists  developed  to  aid  in  the  operation  of  test 
equipment. 

Functionality/Integration  Test  Checklist 
(TCAC-PVD) 


Date; _ 

Scenario:  _ _ _ — — 

Test  Stop  Time  (z): 
Scenario  Stop  Time: 


Test  Start  Time  (z): 
Scenario  Start  Time: 


^mm\ 

POC 

Action 

Event 

Go/No  Go 

Run  PVD 

1 

ETE 

Power  on  the  hp735  monitor. 

Log  on  to  the  hp735  as  hovey. 

2 

ETE 

From  the  xterm  window  that 
appears,  type  pvd,  and  hit  enter. 

Use  this  alias  to  start 

Janus  plan  view  display. 

3 

ETE 

From  the  Janus  plan  view 
display  menu,  verify  the 
parameters  for  the  run: 

Workstation  1 

Terrain  File 

Screen  File 

Symbol  File  3 

Symbol  Size  10 

Terrain  File  Meridian  45 
Exercise  ID  BLANK 

Map  Spheroid  I 

Mode  BLANK 

Terminate  this  Run  N 

and  hit  keypad  enter. 

Use  the  correct  terrain  file 
and  screen  file  for  the 
vignette. 

4 

ETE 

Wait  until  the  PVD  terrain  and 
combat  systems  databases  are 
loaded. 

Last  message:  Opening 
file 

../jads  ete/tm/TSCRN  . 
DAT 

5 

ETE 

Double  click  the 
Analyst_Workstation_WS}  icon 
to  bring  up  the  scenario 
window. 

6 

ETE 

From  the 

Analyst_Workstation_WSl 
scenario  window,  functions 
menu, 

left  click  Draw  CAC. 

7 

ETE 

From  the 

Analyst_Workstation_WSl 
scenario  window,  CAC  File 
menu,  select  the  CAC  file 
number  to  display. 

Left  click  increases  number, 
and  right  click  decreases 
number. 

Left  click  Add  to  display  the 
CAC. 

Places  command  and 
control  overlays  on  the 
scenario  box. 

8 

ETE 

From  the 

Analyst_Workstation_WSl 
scenario  window,  function 
menu, 

left  click  Display. 

Ready  to  receive  and 
display  DIS  PDUs. 

9 

ETE 

i 

From  the 

Analyst_Workstation_WSl 
scenario  window  menu: 

Left  click  any  tick  on  the  zoom 
in/out  menu,  then  select  the 
desired  zoom  point  on  the 
scenario  box. 

Left  click  CAC. 

Left  click  Display. 

Left  click  Clear. 

Used  as  necessary  to 
zoom  in/out  of  the 
scenario  box. 

Used  as  necessary  to  add 
or  remove  the  command 
and  control  overlays 
which  have  been  added  in 
step  7. 

Used  as  necessary  to  start 
or  stop  Janus  plan  view 
display  from  receiving 
PDUs. 

Used  as  necessary  to  clear 
any  text  or  information 
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displayed  on  the  scenario 
box. 


NOTE:  A  particular 
function  is  active  when 
highlighted. _ 


1  aSteri#  POC  .  Action  Event  Go/NoGo 

Stop  PVl 

1 

) 

ETE 

From  the 

Analyst_Workstation_WSl 
scenario  window, 
right  click  End. 

Shuts  down  PVD. 

'  '  ' 

2 

ETE 

Minimize  the 
Analyst_Workstation_WSl 
scenario  window. 

In  the  xterm  that  remains, 
verify  this  message: 

STOP - JANPVD 

Program  Terminated - 

3 

ETE 

From  the 

Analyst_Workstation_WSl 

icon, 

right  click  and  choose  close. 

Closes  the  scenario 
window. 

immm 

POC 

Action 

-''Event  ,  Go/No  Go 

Shut  Down  Test  _ _ _ _  1 

1 

ETE 

Left  click  EXIT  from  the  HP 
VUE  front  panel. 

Signs  off  the  hp735. 

;t=;  .  •  - 

2 

ETE 

Left  click  Continue  logout  from 
the  dialog  box. 

Confirms  desire  to  log 
out. 

V;-  : 

3 

ETE 

Power  off  the  hp735  monitor. 

A1.3  WSMR  Procedures 

This  checklist  is  representative  of  the  individual  site  checklists.  It  incorporates  the  logger 
functionality  and  the  site  specific  actions  required  by  the  opefator(s).  These  are  maintained  by 
the  site  specialists  and  updated  as  changes  are  required. 

Functionality/Integration  Test  Checklist 
(WSMR) 


Date: 
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Scenario:  _ 

Janus  File:  _ _ 

Indy  File:  _ 

Test  Start  Time  (z): 
Scenario  Start  Time: 


Test  Stop  Time  (z): 
Scenario  Stop  Time: 


POC 

Go/No  Go  1 

1  Network  Activation  | 

1 

ETE 

Verify  operation  of  hotlink 

Enables  secure  and 

and 

phone.  If  no  go,  contact  N&E 

unclassified  voice 

N&E 

to  fix  the  network. 

communications. 

2 

ETE 

Verify  that  WSMR  indy  and  the 

Initial  step  in  ensuring 

and 

WSMR  hp715  are  on  the  IADS 

network  is  operational. 

WSM 

R 

ETE  network. 

3 

N&E 

Verity  N&E  has  cleared  and 

Clears  router  interface 

reset  routers. 

cards. 

\  i  •  .  ' ' 

4 

ETE 

Power  on  the  WSMR  monitor. 

Restarts  the  indy. 

Log  on  to  WSMR  as  dislog. 
From  a  Unix  shell  window  as 

su,  run  /scripts/restart. 

5 

ETE 

After  a  successful  restart,  log 

Signon  is  used  for 

on  to  wsmr  as  dislog. 

checking  network 
communications  and 
logging  PDU  data. 

6 

ETE 

From  a  Unix  shell  window  as 

Verifies  that  each  network 

su. 

link  is  operational.  3% 

run  /scripts/ping_test  to  get 

loss  at  Fort  Sill  and  uts  is 

ping  statistics  for  each  remote 
site. 

normal. 
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7 

ETE 

From  a  Unix  shell  window  as 
su,  nm /scripts/checkjtime. 

Displays  the  offset  from 
uts.  Should  be  less  than  1 

ms. 

8 

ETE 

From  a  Unix  shell  window  as 
su  and  at  the  test  controller’s 
direction, 

run  /scripts/run_player  3000 
/disk2Aogfiles/ne_test.log  to 
check  ability  to  send  PDUs  to 
each  remote  site. 

Verifies  sending  2281 

PDUs  and  receiving  the 
same  number  of  PDUs  at 
each  remote  site. 

9 

ETE 

From  a  Unix  shell  window  as 
su  and  at  the  test  controller’s 
direction, 

run  /scripts/dt_logger 

to  check  ability  to 
receive  PDUs  from  a  remote 
site.  At  the  test  controller’s 
direction,  hit  Ctrl-C  to  end  the 
log  file. 

Verifies  receiving  2281 
PDUs  from  a  remote  site. 

10 

ETE 

From  a  Unix  shell  window, 
cd  /usrA,ocal/bin  and 
run  ./display _pdu_rate. 

Select  port  3000  0. 

Left  click  start. 

Verifies  that  PDU_rate  = 

0.  Ensures  that  there 
aren’t  any  DIS 
communications  before 
the  start  of  testing. 
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Go/No  Go ' 


Action 


Event 


Go/NoGo 


Start  Janus 

1  I  I 


ETE 

Power  on  the  c  1 80  monitor. 

Log  on  to  the  cl 80  as  JADS. 

ETE 

From  an  hpterm  window,  type 
janus.exe,  and  hit  enter. 

Use  this  executable  to 
start  Janus. 

ETE 

Left  click  PE  (Program 
Execution)  from  the  Janus  User 
Options  menu. 

Brings  up  the  Program 
Execution  menu. 

ETE 

Left  click  JE  (Janus  Execution) 
from  the  Program  Execution 
mienu. 

First  step  in  defining  the 
scenario. 

ETE 

Type  desired  scenario  number 
for  the  run.  and  hit 

enter. 

Type  run  number  1,  and  hit 
enter. 

Tells  Janus  which 
scenario  to  run. 

ETE 

i 

Hit  enter  again  to  continue. 

Ready  to  continue. 

ETE 

Verily  that  1  is  entered.  Hit 
enter  one  more  time. 

Use  a  normal  run. 

ETE 

From  the  Janus  Runtime 

Screens  menu, 
left  click  11. 

Verify  time  of  day  is  correct  for 
the  vignette,  and  hit  keypad 
enter. 

Verifies  time  of  day. 

ETE 

From  the  Janus  Runtime 

Screens  menu, 

Verifies  that  a  controller 
workstation  has  been 
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left  click  22. 

Verify  that  there  is  a  setup  for: 
WiS  Number  i,and  Side  1, 
and  hit  keypad  enter. _ 


ETE  From  the  Janus  Runtime 
Screens  menu, 
left  click  66. 

Verify  the  DIS  operational 
parameters  for  the  run: 

Janus  side  1 
DIS  side  2 
DIS  COMM  calls/sec 

Units  processed/COMM  call 

Terrain  File  Meridian  (+E)  45 
Heartbeat(s) 


configured. 


Verifies  DIS  parameters. 
Calculate  the  new 
heartbeat  as  follows: 

CxRxH<T 

where 

C  =  calls/sec, 

R  =  units/call, 

H  =  heartbeat,  and 
T  =  total  number  of  units 
in  scenario 


Dead  Reckoning  Threshold 
999 

Site  TRAC’WSMR  23 
Host  CPU  HP  4 
Exercise  JADS-ETE  4 
DIS  version  transmit  4 
DIS  version  receive  4 


and  hit  keypad  enter. _ 


Left  click  JJ  (Begin  Janus) 
from  the  Janus  Runtime 
Screens  menu. 


Wait  until  the  Janus  scenario 
loads.  Verify: 

Scenario  number _ 

Total  number  of  units 


Double  click  the  sidel  icon  to 
bring  up  the  scenario  window. 


Loads  the  Janus  scenario. 


This  brings  up  the 
scenario  window  which 
allows  a  Janus  operator  to 
interact  (game)  the 


exercise. 
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Run  Scenario 

1  1  EXE 


Action 


From  the  sidel  scenario 

window, 

left  click  D/S. 


EXE 

From  the  sidel  scenario 
window, 

left  click  SFARX. 

EXE 

Minimize  the  Janus  scenario 
window  (sidel).  Xype  ir  in  the 
Janus  window,  and  hit  enter. 

EXE 

Xype  n  and  hit  enter. 

EXE 

Hit  enter  again. 

EXE 

Double  click  the  Janus  scenario 
window  (sidel). 

EXE 

Verify  that  loggers  are  logging. 

'■"Event 


DIS  button  highlights. 
Opens  DIS 
communications . 


First  step  in  running  a 
Janus  scenario. 


Ready  to  continue  the 
Janus  run. 


No  planned  save. 


Default  checkpoint 
frequency. 


Verifies  scenario 
movements  and  a  running 
time  of  day  counter. 


<5o/No  Go 


POC 

Action 

Go/No^^ 

Stop  Scenario 
1  I  ET 


EXE 

From  the  sidel  scenario 

window, 

left  click  DIS. 

DIS  button  unhighhghts. 
Closes  DIS 
communications. 

EXE 

From  the  sidel  scenario 
window, 

right  click  ADM/V. 

Brings  up  options  menu. 

EXE 

Left  click  EJ  (End  Janus). 

Quits  the  scenario  run. 

EXE 

Right  click  2  times. 

Completely  closes  Janus. 

EXE 

Left  click  EXIT  from  the  HP 
VUE  front  panel. 

Sign  off  the  hp715. 

■mitH 

Action  :  :  '■  Event- --^Go/No GcH 

Shut  Dov 

1 

m  Test 
ETE 

Power  off  the  cl 80  monitor, 
and  shutdown  CPU. 

2 

ETE 

and 

N&E 

Make  sure  that  JADS  N&E 

FTP 

/dixklAogfiles/ 

mr.log 

back  to  JADS  and  place  the  file 
in 

/usr/testdata2/logs/ete/DDMM 

IT 

Ensures  data  integrity. 

This  file  will  be  analyzed 
by  JADS  analysts. 

3 

ETE 

Power  off  the  wsmr  monitor. 

4 

ALL 

After-action  review 

■  ,:,x  ■  •  .  ■; 
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Appendix  C  -  TEXCOM  Test  Simulation  Development  Report 
Cl.O  Introduction 

The  U.S.  Army  Test  and  Experimentation  Command  (TEXCOM)  Technology  Laboratory  (TTL) 
was  selected  to  develop  the  scenario  and  provide  scenario  support  for  the  Joint  Advanced 
Distributed  Simulation  (JADS)  Joint  Test  Force  End-to-End  Test.  All  elements  of  TTL, 
government  and  TTL  support  contractors  assisted  in  supporting  the  JADS  project  at  appropriate 
times  in  the  process. 

Cl.l  Report  Requirement 

Task  Number  9200.1,  Fiscal  Year  (FY)  99  JADS  Work  Authorization  Order,  delineates  the 
requirement  for  this  report. 

C1.2  Scenario  Development  Environment 

TTL  support  analysts  assisted  JADS  test  team  members  with  test  simulation  design  from  early 
concept  development  through  final  simulation  preparation.  The  scenario  support  analysts 
assisted  with  adjustments  to  the  simulation  and  facilitated  use  of  the  simulation  tools  during  the 
live  test  phases.  The  active  exchange  among  test  team  members  and  the  TTL  scenario  support 
cell  fostered  full  communication  and  a  common  understanding  of  test  scenario  issues.  This 
positive,  professional  environment  created  common  understanding  and  trust  and  kept  the 
scenario  development  and  support  focused. 

C1.3  Report  Objectives 

-  Record  the  scenario  development  process  as  planned  and  as  evolved. 

-  Assess  the  scenario  development  process  and  recommend  improvements  for  use  if  there  is  a 
future  related  requirement. 

-  Examine  the  scenario  development  process  to  determine  what  parts  may  be  assisted  with 
automation. 
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C2.0  Test  Simulation  Support  Methodology 
C2.1  Set  Simulation  Conditions 
C2.1.1  Develop  The  Road  to  War 

The  environment  and  circumstance  for  a  simulated  conflict  was  set  in  The  Road  to  War 
document.  Users  were  provided  background  for  events  leading  to  the  notional  conflict  and  a 
description  of  the  ensuing  battle.  The  scenario  was  adapted  from  the  U.  S.  Army  Command  and 
General  Staff  College  (CGSC)  Common  Teaching  Scenario  Southwest  Asia,  dated  April  1992,  as 
modified  by  Training  and  Doctrine  Command  (TRADOC). 

C2.1.2  Prepare  Simulation  Support  Documents 

Several  key  documents  were  developed  to  facilitate  and  manage  corps  battle  simulation  (CBS) 
game  play.  They  were  necessary  to  define  the  forces  played  and  to  provide  a  sound,  structural 
basis  for  adaptations  and  adjustments  that  would  occur  during  the  use  of  the  scenario  in  the 
various  test  phases. 

C2.1.3  Force  Hierarchy 

The  red  force  hierarchy  was  adapted  from  the  Advanced  Tactical  Command  and  Control  System 
(ATCCS)  common  scenario  and  developed  in  detail  sufficient  to  support  the  test.  The  blue  force 
hierarchy  included  U.  S.  HI  Corps  units  plus  coalition  forces  and  an  additional  U.  S.  division,  all 
represented  as  brigade-size  icons.  Red  forces  were  shown  both  as  l^ge  units  and  as  units  broken 
down  to  smaller  fighting  elements  as  needed  to  depict  the  resolution  necessary  for  the  test.  The 
precise  delineation  of  the  force  hierarchy  was  so  each  unit  could  be  tagged  and  so  all  movements 
and  actions  could  be  tracked  in  the  game. 

C2.1.4  Supporting  Units 

All  units  had  to  have  a  supporting  unit  specified.  The  CBS  logistical  supply  process  requires  the 
supporting  units  to  serve  as  flow  points  for  resources  to  their  respective  units  as  may  be  needed. 

C2.1.5  Unit  Templates 

Unit  templates  were  prepared  for  all  different  unit  types  and  sizes.  The  templates  showed  the 
unit  personnel  stmcture,  the  equipment  list  and  associated  unit  resources.  Each  template  was 
coded  so  it  could  be  assigned  to  all  like  units. 
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C2.1.6  Start  Locations 


All  units  had  to  be  assigned  starting  locations  at  game  start  to  set  the  battle  picture.  The  units 
were  arrayed  within  the  control  lines  of  the  respective  forces  as  shown  in  both  red  and  blue 
operations  orders. 

C2.1.7  Unit  Strength  Percentage 

All  units  were  set  at  a  force  strength  that  might  be  reasonably  expected  for  the  start  of  the 
conflict.  Once  the  conflict  started,  strength  levels  changed  through  normal  consumption, 
resupply  or  battle  attrition.  Resupply  was  conducted  by  established  CBS  game  rules. 

C2.1.8  Prepare  Operations  Orders  for  Red  and  Blue  Forces 

Operations  orders  (OPORDs),  including  selected  annexes,  were  written  for  both  the  opposing 
and  friendly  forces.  They  were  used  for  orientation  to  the  anticipated  operation  and  for  a 
common  understanding  of  the  battlefield  parameters  in  preparation  for  developing  the  CBS  game 
tape. 

C2.1.9  Build  Corps  Battle  Simulation  Game  Tape 

The  CBS  game  tape  was  built  in  the  TTL  in  December  1997  and  January  1998.  The  team 
assembled  to  build  the  game  tape  was  CBS-trained  TTL  support  contractors  and  professional 
CBS  trainers/coaches/players  from  IH  Corps  Battle  Simulation  Center.  All  the  support 
documents  were  assembled  as  working  manuals  and  provided  to  all  team  members.  The 
TEXCOM  team  members  were  assigned  to  manage  blue  forces  during  the  tape  build.  The  CBS 
team  members  were  assigned  to  manage  the  red  forces.  The  red  forces  would  be  deployed  in 
finer  resolution,  as  the  focus  of  the  test  would  be  on  the  opposing  forces.  Some  of  the  test 
segments  would  focus  on  rear  area  logistics  and  others  on  the  movements  of  committed  line 
combat  units. 

C2.1.10  Train  CBS  Workstation  Operators 

All  the  workstation  operators  were  versed  in  CBS  game  procedures.  The  limited  training  was 
tailored  to  individual  workstation  assignments.  Operators  were  assigned  to  maneuver  combat 
units,  artillery,  logistics  forces  and  air  defense.  With  areas  of  responsibilities  assigned,  all  the 
operators  studied  support  documents  and  their  respective  tasks  so  they  could  play  the  game  in  a 
coordinated  manner. 

C2.1.11  Deploy  Forces  in  CBS 

All  the  forces  for  blue  and  red  programmed  into  the  game  had  to  be  moved  to  the  planned  start 
locations.  With  the  number  of  units  that  had  to  be  deployed,  the  initial  force  deployment  took  the 
major  part  of  a  week. 
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C2.1.12  Validate  Force  Structure 

After  forces  were  assigned  to  preplanned  start  locations,  all  the  units  were  checked  for  proper 
military  tables  of  organization  and  equipment.  Tactical  locations  of  units  were  double  checked 
for  specific  location  and  for  tactical  relationship  with  adjacent  units. 

C2.1.13  Compile  128-Hour  CBS  Tape 

With  all  parties  prepared  and  forces  positioned  for  game  start,  the  tape  build  began.  The  CBS 
game  play  was  governed  by  a  detailed  force  synchronization  matrix  that  showed  what  combat 
maneuver  was  expected  at  what  time.  It  also  showed  the  expected  actions  of  supporting  forces  of 
all  varieties.  This  matrix  served  as  a  script  for  the  tape  build.  Some  lead-time  start-up  was 
needed  each  day.  With  that  consideration,  the  team  worked  to  build  from  six  to  eight  hours  of 
finished  CBS  game  tape  each  day.  Downtime  was  kept  to  a  minimum,  and  the  build  progressed 
as  planned. 

C2.1.14  Monitor  Intelligence  Collection  Devices 

The  Joint  Exercise  Support  System  Intelligence  Module  (JIM)  was  used  to  collect  intelligence  of 
all  types  from  the  game  as  the  game  build  was  in  progress.  A  special  JIM  team  linked  its 
collection  program  to  CBS  and  provided  a  parallel  stream  of  intelligence  messages  that  would 
normally  be  expected  during  actual  tactical  operations.  Those  messages  were  prepared  for 
dissemination  during  the  actual  test  to  feed  the  test  unit  player  cell  with  a  realistic  flow  of 
intelligence.  Those  messages  would  be  combined  with  information  from  the  test  Joint 
Surveillance  Target  Attack  Radar  System  (Joint  STARS)  aircraft  to  formulate  intelligence 
targeting  recommendations  to  the  tactical  battle  captains.  The  intelligence  information  presented 
through  the  JIM  system  was  less  than  100  percent  of  what  was  actually  happening,  zis  would 
normally  be  expected  in  an  actual  conflict. 

C2.1.15  Validate  CBS  Tape 

The  CBS  game  tape  was  reviewed  in  detail.  The  review  was  conducted  to  search  for  anomalies 
in  the  tape  to  ensure  no  errors  had  been  introduced  that  placed  units  in  illogical  locations  and  that 
there  were  no  gaps  in  JIM  sensor  coverage.  Several  gaps  were  found  in  the  JIM  products  and 
data  were  developed  to  fill  those  gaps. 

C2.1.16  Prepare  Vision  XXI  to  Facilitate  Game  Truth  Capture 

Vision  XXI,  developed  by  Tapestry  Solutions,  Inc.,  is  a  real-time  exercise  monitoring  tool  that 
provides  a  highly  effective  visual  interface  to  track  current  and  historical  battle  conditions.  It  has 
the  capability  to  watch  individual  unit  movement  and  resource  status.  The  128-hour  game  tape 
was  loaded  into  a  Vision  XXI  workstation.  By  using  the  historical  mode,  analysts  were  able  to 
examine  all  phases  of  the  portrayed  battle  at  will  in  varied  force  resolution  and  determine 
locations  and  times  of  activity  that  could  best  be  used  for  test  purposes. 
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C2.2  Establish  Ground  Reference  Coverage  Areas  (GRCAs) 

The  TTL  scenario  support  team  lead  and  the  JADS  project  officer  jointly  selected  GRCAs.  First, 
Joint  STARS  system  experts  and  the  test  team  selected  orbital  flight  paths.  They  considered 
potential  threats  against  the  aircraft  and  worked  to  select  optimal  tracks  to  cover  the  test  terrain 
area.  Then  the  GRCAs  were  plotted  as  they  considered  appropriate  orbit  tracks  for  the  aircraft 
and  the  various  types  of  battlefield  action  that  would  best  be  used  to  support  the  different  test 
objectives.  Three  different  orbits  were  set  and  the  associated  GRCAs  were  plotted  so  there  could 
be  coverage  of  different  areas  of  the  battlefield.  The  GRCAs  were  oriented  parallel  to  the  flight 
orbits.  Both  CBS  and  Janus  game  portrayal  boxes  are  north-south,  east-west.  The  GRCAs  were 
set  at  an  angle  to  the  game  boxes.  The  resulting  misalignment  increased  the  complexity  of 
converting  data  from  CBS  to  Janus. 

C2.3  Identify  Vignette  Periods 

Nine  different  six-hour  periods  were  selected  to  focus  on  different  battle  phases  and  different 
types  of  combat  force  and  combat  support  activity.  The  synchronization  matrix  used  during  the 
game  tape  build  helped  as  a  guide,  but  to  fully  examine  activity  in  detail,  the  total  recorded  game 
had  to  be  examined.  The  CBS  game  tape  was  loaded  on  the  Vision  XXI  workstation.  With  this 
setup,  analysts  could  step  through  the  game  in  time  increments  or  examine  game  activity  in 
specified  time  blocks.  With  a  list  of  activity  types  best  suited  for  test  purposes,  appropriate  six- 
hour  time  blocks  were  identified. 

C2.4  Provide  CBS  Data  for  Conversion  to  Janus 

The  CBS  system  is  an  aggregate-based  simulation.  Symbols  of  military  units  displayed  represent 
all  the  people,  equipment  and  other  resources  associated  with  that  unit.  Status  of  units  can  be 
determined  by  querying  the  CBS  game.  The  Joint  STARS  sensors  are  built  to  see  individual 
pieces  of  equipment  not  unit  icons.  Janus  is  an  entity-based  game;  it  sees  individual  pieces.  For 
the  Joint  STARS  sensors  to  function  in  relationship  to  the  test  scenario,  the  data  about  the  forces 
played  needed  to  be  converted  from  an  icon  base  to  an  entity  base.  The  movement  tables  that 
showed  movements  of  all  the  red  forces  throughout  the  recorded  CBS  game  were  extracted  and 
taken  to  Janus  programmers  at  the  U.S.  Army  Training  and  Doctrine  Command  Analysis  Center 
(TRAC),  White  Sands  Missile  Range  (WSMR),  New  Mexico. 

The  process  of  preparing  the  data  for  the  Janus  programmer  involved  several  people  and  took 
several  weeks  to  complete.  A  TTL  programmer  worked  with  the  CBS  database  and  pulled 
information  in  ten-minute  intervals  on  all  units  that  moved.  The  Janus  programmer  needed  all 
unit  data  arranged  hierarchically  showing  initial  movement  and  subsequent  movements.  Other 
data  sets  reflected  logistics  deltas,  unit  location  deltas,  radar  unit  deltas,  air  defense  deltas  and 
unit  equipment  deltas.  After  identifying  all  unit  data  in  the  respective  GRCAs,  the  data  were 
converted  to  rich  text  format  so  they  would  be  transportable. 
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The  TTL  scenario  team  lead  worked  directly  with  the  Janus  programmers  to  begin  the  process  of 
converting  that  portion  of  the  CBS  game  data  that  fell  within  the  respective  GRCA  for  each  of 
the  nine  six-hour  vignettes.  After  a  period  of  familiarization  and  the  completion  of  the  first  and 
second  vignettes,  the  TTL  scenario  team  lead  left  the  conversion  to  the  Janus  programmers.  The 
TTL  continued  to  provide  support  to  the  Janus  programmers  by  telephone. 

C2.5  Prepare  JIM  Data  for  Test  Support 

The  JIM-produced  intelligence  message  traffic  was  extracted  and  printed  from  the  CBS  game 
tape  in  the  six-hour  segments  corresponding  to  the  series  of  vignettes.  The  messages  were  sorted 
by  time  blocks  and  prepared  for  passing  to  the  participating  test  unit  intelligence  cell.  The 
message  traffic  would  be  used  to  track  red  forces  in  the  best  resolution  possible  and  would  be 
combined  with  test  Joint  STARS  sensor  data  to  formulate  target  nominations. 

C2.6  Facilitate  Test  Intelligence  Traffic  Flow 

The  intelligence  message  traffic  was  passed  to  the  test  participants  at  the  predetermined  times. 
The  facilitators  also  helped  deconflict  the  CBS  aggregate  location  data  with  the  Janus  entity- 
based  location  data.  The  deconflicted  data  were  used  to  improve  the  realistic  response  to  artillery 
action  against  Janus  depicted  targets. 

C3.0  Scenario  Development  Process  Improvement  Recommendations 
C3.1  Lessons  Learned 

The  JADS  scenario  and  scenario  support  materials  were  developed  from  the  best  information 
from  the  beginning  to  the  end  of  the  project.  The  experiences  gained  yielded  several  key  lessons 
learned. 

-  The  order  of  the  scenario  development  and  support  process  is  important.  Much  of  the 
scenario  development  is  a  building  block  process.  If  any  step  is  out  of  order,  omitted  or 
slighted,  the  final  product  can  be  seriously  degraded.  Efforts  to  go  back  and  change  may 
affect  the  work  completed  after  the  point  of  correction.  A  supporting  example  from  the 
JADS  test  would  be  that  the  GRCAs  were  plotted  after  the  scenario  development  process  was 
well  along,  ff  the  GRCAs  and  the  associated  Joint  STARS  aircraft  flight  paths  had  been  set 
early  in  the  scenario  development  process,  the  combat  and  combat  support  activity  in  CBS 
could  have  been  deployed  and  maneuvered  in  the  areas  that  could  be  observed  by  the  Joint 
STARS  sensors. 

—  The  current  conversion  process  from  aggregate  simulation  to  entity  simulation  can  be  done, 
but  the  process  is  lengthy  and  largely  sequential.  Adequate  time  and  resources  must  be 
planned  to  support  the  data  conversion. 
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-  Early  delineation  of  test  and  scenario  development  requirements  helps  the  whole  process 
efficiency.  The  scenario  can  be  properly  developed  to  support  initially  stated  requirements. 
New  or  changed  test  requirements  that  surface  after  critical  development  points  may  not  be 
fully  serviced  in  the  scenario  development  because  of  time  and  resource  constraints. 

-  The  planning  timeline  should  allow  for  inevitable  adjustments  as  the  test  develops. 

C3.2  Revise  Process  Order 

If  all  relevant  scenario-related  planning  information  could  be  gathered  and  resources  identified 
before  beginning  the  development  process,  the  following  adjusted  order  would  be  appropriate: 

-  Key  on  test  objectives  at  the  beginning  of  the  scenario  development  process. 

-  Set  supporting  scenario  objectives.  Build  to  fully  support  test  and  allow  for  full  examination 
of  systems  under  test. 

-  Develop  The  Road  to  War  to  delineate  desired  scenario  environment. 

-  Define  sensor  coverage  area.  Establish  GRCA. 

-  Consider  coverage  area  of  sensors  when  CBS  and  Janus  simulation  boundaries  are  set. 

-  Prepare  simulation  support  documents  to  manage  CBS  game  play. 

-  Set  the  force  hierarchy.  Determine  friendly  forces  to  be  deployed  and  assess  the  desired 
opposing  force  stmcture  needed. 

-  Assign  supporting  units. 

-  Validate  force  structure. 

-  Prepare  unit  templates. 

-  Set  unit  start  locations. 

-  Set  unit  strength  percentages. 

-  Prepare  red  and  blue  operations  orders. 

-  Construct  synchronization  matrix  to  coordinate  scenario  build. 

-  Plan  simulation  to  focus  scenario  activity  for  each  set  of  vignette  objectives. 
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-  Structure  matrix  to  serve  as  a  script  during  CBS  scenario  tape  build. 

-  Train  CBS  workstation  operators. 

-  Build  CBS  game  tape. 

-  Set  and  monitor  intelligence  collection. 

-  Validate  CBS  tape. 

-  Prepare  Vision  XXI  and  game  truth  capture. 

-  Provide  CBS  data  for  conversion  to  Janus. 

-  Prepare  to  support  test  team  during  actual  event. 

C4.0  Assess  Process  Automation  Potential 

Within  the  resources  and  time  allotted,  an  assessment  was  made  to  determine  where  automation 
could  assist  in  developing  a  similar  or  larger  scale  simulation-based  event.  People  who  were 
experienced  in  the  development  and  operations  of  CBS  and  Janus  were  consulted.  Several  others 
who  are  involved  in  the  development  and  integration  of  future  simulation  support  programs  were 
interviewed.  People  in  the  military  testing  coimnunity  and  active  military  operations  were  also 
queried.  The  following  observations  were  derived  from  that  assessment. 

C4.1  Observations 

-  The  present  tools  used  to  develop  a  simulated  environment  combing  current  aggregate 
simulations  (such  as  CBS)  and  entity  simulations  (such  as  Janus)  are  inadequate  to  support 
real-time  interaction  between  the  two  types  with  any  fidelity.  These  simulations  were 
designed  for  specific  purposes  and  were  not  intended  to  be  interactive.  The  current  process 
used  for  JADS  resulted  in  fixed  scenarios  in  Janus.  This  process  does  not  allow  for  real-time 
interaction  between  a  live  CBS  game  and  a  live  Janus  game. 

-  The  related  developmental  efforts  are  formative  and  incomplete  at  this  time. 

-  There  are  no  interface  tools  presently  developed  to  successfully  support  real-time  interaction 
of  the  subject  systems. 

-  The  Joint  STARS  aircraft  platform  and  its  associated  sensors  cannot  be  actively  integrated 
into  large-scale  simulations  such  as  those  supporting  integrated  Automated  Battle  Control 
System  tests  or  U.S.  Army  corps-level  warfighters  unless  entity-level  resolution  of  forces  is 
depicted. 
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C4.2  Focus  for  Potential  Automation  Support 

The  scenario  development  process  used  in  the  JADS  test  focused  on  simultaneous  use  of  CBS 
aggregate  and  Janus  entity-based  simulation  systems.  If  the  test  and  operations  communities 
choose  to  further  develop  this  type  of  fixed  scenario  or  to  develop  a  larger  more  interactive 
scenario  support  set  to  facilitate  large-scale,  free-play  operations,  automation  must  be 
incorporated  into  the  process. 

The  need  for  automation  should  be  considered  in  two  categories. 

-  Automation  support  to  the  basic  scenario  development  process  used  for  the  JADS  End-to- 
End  Test.  This  process  resulted  in  fixed  Janus  scenario  vignettes. 

-  Automation  applied  to  work  the  CBS  to  Janus  or  aggregate-based  simulation  to  entity-based 
simulation  translation  problem  that  would  enable  real-time  interaction  between  CBS  and 
Janus. 

C4.2.1  Basic  Scenario  Automation  Support 

Several  ad  hoc  automation  tools  were  developed  during  the  scenario  development.  The 
preparatory  documentation  and  databases  were  developed  within  personal  computer  office  tools. 
Much  of  the  data  manipulation  after  the  CBS  tape  was  built  was  done  within  the  CBS  VAX  host 
computers.  The  Vision  XXI  tool  described  earlier  was  invaluable  as  analysts  searched  the 
prepared  game  for  activity  that  would  support  the  test  best.  Vision  XXI  was  also  used  as  an 
automated  monitor  tool  during  the  actual  test  phases. 

Though  automation  support  was  integral  to  the  scenario  development  process,  the  human  effort 
required  to  work  through  many  of  the  steps  the  first  time  begs  more  automation  support. 
Developing  macros  or  other  small  programs  to  speed  the  development  process  can  boost  many  of 
the  scenario  development  steps.  If  the  revised  process  order  of  paragraph  C3.2  is  followed,  an 
event  planner  should  apply  automated  support  to  as  many  of  these  steps  as  possible. 

C4.2.2  Aggregate  to  Entity-Based  Simulation  Automation  Support 

The  aggregate  to  entity  translation  developed  for  the  JADS  End-to-End  Test  consisted  of 
converting  pretaped  CBS  data  to  Janus  format.  Given  adequate  resources,  programmers  should 
be  able  to  develop  effective  translator  tools  to  use  for  this  situation.  Assuming  the  development 
of  the  tools,  upgrades  to  the  current  two  simulation  support  systems  and  any  follow-on  systems 
would  have  to  be  resolved  with  a  common  configuration  management  program.  Without 
common  configuration  management,  any  translator  development  would  quickly  become  out  of 
date. 

If  an  aggregate  to  entity  translator  that  refined  and  translated  precanned  scenario  data  was 
successfully  developed,  the  product  from  that  translator  would  be  limited  in  use.  Possible  uses 
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might  be  checks  or  tests  of  isolated  activities.  Most  large-scale  system  testing  and  operational 
training  involves  free-play  and  allows  battle  managers  to  maneuver  force  on  force.  The  type  of 
translator  discussed  so  far  may  help  with  pre-event  systems  checks  prior  to  a  primary  free-play 
event.  However,  that  translator  would  not  support  dynamic  free-play  where  movement  and 
location  data  were  not  fixed  for  the  whole  scenario. 

The  larger  problem  set  of  developing  translators  between  aggregate  and  entity-based  simulation 
to  support  dynamic  military  operations  is  complex.  The  cost  efficiency  of  such  an  endeavor 
would  need  to  be  analyzed  in  a  separate  study. 

C5.0  Report  Summary 

The  TTL  and  the  attached  contractor  staff  supported  the  JADS  ETE  Test  by  providing  scenario 
preparation  and  support  from  concept  stage  through  test  execution.  The  process  employed 
demonstrated  what  could  be  done  to  provide  a  large-scale,  aggregate  test  environment  and  a 
limited  focus  at  the  entity  level  for  fixed  scenario  periods  or  vignettes.  The  demonstration  did 
not,  however,  answer  requirements  for  large-scale  depiction  of  entity  level  combat  forces.  The 
process  the  TTL  used  could  be  reduced  in  complexity  and  time  required  with  the  development  of 
automation  tools  to  use  on  various  stages  of  the  process.  The  larger  issue  of  developing  a  large- 
scale,  real-time  dynamic  interface  between  CBS  and  Janus  remzdns  an  open  issue.  Further  study 
to  analyze  the  need  and  cost  efficiency  of  such  a  large-scale,  dynamic  CBS  to  Janus  interface 
should  follow. 
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Appendix  D  -  Janus  Support  for  JADS  Joint  Test  and  Evaluation  (JT&E) 

Dl.O  SimulationSupportto  JADS  JT&E 
Dl.l  Constructive  Simulation  Selection  Process 

To  generate  the  “notional  corps”  rear  area,  a  constructive,  entity-level  simulation  was  required. 
These  requirements  were  identified  as  evaluation  criteria  of  the  capability  of  various  existing 
simulations  to  support  the  JADS  End-to-End  (ETE)  Test: 

•  Capable  of  simulating  or  of  being  modified  to  simulate  at  least  5000  distinct  entities  with  at 
least  twenty-five  percent  moving  in  a  reasonable  and  affordable  manner 

•  Capable  of  issuing  distributed  interactive  simulation  (DIS)  2.0.4  entity  state  protocol  data 
units  (ESPDUs)  that  describe  each  entity  simulated 

•  Capable  of  receiving  and  acting  upon  ESPDUs,  fire  protocol  data  units  (PDUs),  and 
detonation  PDUs 

•  Capable  of  running  for  at  least  eight  hours  with  human  intervention  as  required 

•  Capable  of  running  at  or  representing  real-time  actions 

•  Capable  of  using  a  terrain  database  at  least  200  kilometers  (km)  by  200  km  and  based  on 
National  Imagery  and  Mapping  Agency  products 

•  Must  have  a  verification  and  validation  (V&V)  history,  an  accreditation  history  for  analysis, 
be  under  configuration  control,  and  be  well  documented 

•  Capable  of  reasonably  representing  entity  maneuvers,  including  stopping  and  turning 

•  Capable  of  representing  the  effects  of  a  bombing  or  missile  attack  on  the  entities  represented 
in  an  acceptable  and  credible  manner 

Table  1  summarizes  the  review  of  six  candidate  simulations.  This  analysis  indicated  that  none  of 
the  constructive  simulations  evaluated  would  meet  the  ETE  Test  requirements  without 
modification  and/or  further  development.  JADS  selected  the  Janus  simulation  as  the  best 
candidate  for  upgrading  to  meet  its  requirements.  Janus  is  under  the  configuration  control  of  the 
U.S.  Army  Training  and  Doctrine  Command  (TRADOC)  Analysis  Command  (TRAC),  White 
Sands  Missile  Range  (WSMR)  and  the  U.S.  Army  Simulation,  Training,  and  Instrumentation 
Command  (STRICOM).  JADS  entered  into  an  agreement  with  TRAC-WSMR  to  modify  Janus, 
as  appropriate,  to  be  able  to  represent  at  least  5000  entities.  The  capability  for  entity-level  data 
generated  by  Janus  to  be  converted  into  ESPDUs  was  made  possible  by  TRAC-WSMR 
developing  an  interface  for  that  purpose. 

D1.2  ETE  Test  Phase  2 

Phase  1  concluded  with  the  successful  definition  and  refinement  of  an  advanced  distributed 
simulation  (ADS)  synthetic  environment.  The  intent  of  Phase  2  was  to  evaluate  the  utility  of  that 
ADS  synthetic  environment  to  support  developmental  test  and  evaluation  and  early  operational 
test  and  evaluation  of.  a  command,  control,  communications,  computers,  intelligence, 
surveillance  and  reconnaissance  (C4ISR)  system  in  a  laboratory  environment. 
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Each  test  activity  required  the  development  of  one  or  more  scenarios  designed  to  meet  the  test 
objectives.  Scenarios  designed  for  developmental  testing  (DT)  are  based  on  the  system 
specifications  of  the  test  system  and  require  careful  analysis  and  testing  to  ensure  the  DT 
objectives  are  met.  Scenarios  developed  for  operational  testing  (OT),  however,  are  based  on  a 
large-scale  scenario  and  are  focused  more  on  operational  realism  than  specification  accuracy. 
JADS  developed  two  scenarios  for  verification  and  validation,  12  scenarios  for  DT,  and  TRAC- 
WSMR  developed  five  scenarios  for  OT.  Note  that  OT  scenarios  were  based  on  an  expanded 
version  (128  hours)  of  a  U.S.  Army  Test  and  Experimentation  Command  (TEXCOM)  scenario. 
The  Road  to  War,  that  has  been  used  to  test  components  of  the  Army  Tactical  Command  and 
Control  System.  JADS  selected  nine  6-hour  vignettes  from  that  scenario,  five  of  which  were 
implemented  in  Janus. 
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Table  D-1.  Comparison  of  Wargaming  Constructive  Simulations 


JANUS 

EAGLE 

CBS 

Brigade  Battalion 
Simulation 

11111 

STAGE 

Operating 

System 

OpenVMS, 

HP-UX,  SunOS 

Sun  OS 

OPS  5 

OpenVMS 

OpenVMS 

Silicon  Graphics  IRIX 

Application 

training  and  combat 
development 

combat 

development 

training 

training 

training  and 

doctrine 

development 

demo 

Number  of 
Entities 

1200  total 
upgradeable  to 

9999 

division  to 
corp-level 
unit 

representation 

15,000  -  20,000  unit 

brigade-level 

representation 

brigade  game  plays 
brigades  - 
using  20  MODS  AFs 
-2000 

600x600 

lots  at  a  cost 

DIS 

Compliant? 

Sep-94 

yes, 

through  SAFOR 

aggregate-level 
simulation  protocol 
to  DIS, 
date  unknown 

yes,  through 

MODSAF 

will  be  when 
combined  with 
Janus 

no 

Cost  to 

Make  DIS 

N/A 

N/A 

N/A 

N/A 

N/A 

unknown 

Battle 

Decision 

Intelligence 

None 

MODSAF 

human-in-the- 
loop  &  expert 
system 

no  -  scripts 

Real  Time 

event  based 
but  events  run 
at  real  time 

SAFOR 

unknown 

MODSAF 

yes 

yes,  depending  on 
processor 

Minimal 

Computer 

Processing 

Unit 

16  MB 

48  MB 

128  MB +  16  MB 

Multiple  (up  to  10 

Micro  VAX's) 

16  MB 

depends  on 
number  of  units  - 
minimal,  Iris 

Terrain 

DMA 

DMA 

HEX? 

laser  disk  video 
displayed 

DMA 

? 

Database 

SWA 

SWA 

SWA 

not  SWA-several 

SWA 

limited  SWA 

Size 

lOOOxlOOOcells 

resolution- 
dependent  size 
upgradeable  to 
400x400km 

corps 

corps  + 

lO.OOOsqkm  to 

30,000sq  km 

100x1 00km 
upgradeable  to 
400x400km 

? 

Timing 

can  play  real  time 

real  time  with 
caution,  5  minute 
interactions 

real  time 

real  time 

can  play 
real  time 

can  play  real  time 

Interactive 

yes 

yes  (SAFOR) 

1 

yes  (MODSAF) 

yes 

by  rewriting  scripts 

Kill 

Assessment 

yes 

yes 

yes 

yes 

yes 

yes 

Data  Flow 
Format 

can  mirror 

can't  mirror 

can’t  mirror 

can't  mirror 

can’t  mirror 

can't  mirror 

Ownership 

TRAC 

STRICOM 

Combined  Arms 
Center  training 

going  to  TRAC- 
WSMR 

proprietary 

V&V 

yes  -  TRAC 

no  -  in  process 

no  -  in  process? 

no  -  status  unknown 

no 

no  -  none  planned 

DMA  =  Defense  Mapping  Agency  km  =  kilometer  MB  =  megabytes 

MODSAF  =  modular  semiautomated  forces  SAFOR  =  semiautomated  forces  sq  =  square 

STRICOM  =  U.S.  Army  Simulation,  Training  and  Instrumentation  Command  SWA  =  Southwest  Asia 


At  the  conclusion  of  the  Phase  2  DT,  the  OT  which  used  the  entire  ETE  Test  synthetic 
environment  to  include  constructive,  virtual,  and  live  participants  began.  Janus  was  used  to 
provide  constructive  targets  for  the  Joint  Surveillance  Target  Attack  Radar  (Joint  STARS)  sensor 
being  simulated  by  the  Virtual  Surveillance  Target  Attack  Radar  System  (VSTARS).  VSTARS 
radar  reports  were  sent  to  the  E-8C  workstations  in  a  Northrop  Grumman  laboratory  and  to  a 
light  ground  station  module  (LGSM)  located  at  Fort  Hood  using  the  surveillance  control  data  link 
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(SCDL)  interface.  The  LGSM  was  in  support  of  a  subset  of  the  tactical  analysis  cell  (TAG)  that 
merged  Joint  STARS  data  with  other  intelligence  sources  to  analyze  the  battlefield  and  select 
candidate  targets  for  the  Army  Tactical  Missile  System  (ATACMS).  The  candidate  targets  were 
sent  to  the  Depth  and  Simultaneous  Attack  Batde  Lab  (D&SA  BL)  at  Fort  Sill,  Oklahoma,  where 
they  were  analyzed  and  a  doctrinally  correct  artillery  command  and  control  was  simulated  to 
determine  if  an  ATACMS  would  be  fired  at  one  or  more  of  the  candidate  targets  Messages 
describing  the  firing  and  detonation  of  the  missile  were  sent  back  to  Janus  for  assessment  of 
damage.  Data  were  also  collected  to  support  the  evaluation  of  operational  measures  of 
effectiveness  and  to  allow  JADS  to  evaluate  the  utility  of  the  environment  to  support  operational 
testing. 


D2.0  Enhancement  of  Janus  Capabilities  to  Support  the  ETE  Test 
D2.1  Background 

Janus  is  a  constructive,  interactive,  computer-based  simulation  of  combat  operations  conducted 
by  platoons  through  brigades.  The  original  Janus  simulation  began  in  the  late  1970s  at  Lawrence 
Livermore  National  Laboratories  to  provide  commanders  with  interactive  control  of  a  combat 
simulation  in  the  presence  of  nuclear  effects.  In  1983,  TRAC-WSMR  obtained  Janus  and 
enhanced  it  as  a  high-resolution  simulation  to  support  analysis  for  Army  combat  development. 
In  1991,  TRADOC  selected  Janus  for  training  at  the  company  and  platoon  levels.  Janus  has  also 
been  used  by  the  Command  and  General  Staff  College  to  train  new  battalion  and  brigade 
commanders  on  the  principles  of  synchronized  combined  arms  operations. 

Janus  is  a  suite  of  programs  made  up  of  more  than  200,000  lines  of  FORTRAN  code.  These 
programs  produce  an  environment  that  allows  the  user  to  set  up  and  execute  a  desired  battle. 
Terrain,  performance  data,  forces,  unit  sjnnbols,  and  command  and  control  overlays  are  typical 
entities  that  can  be  developed  and  edited  to  suit  the  user.  Interactive  graphics  programs  such  as 
the  Janus  analyst  workstation  (JAAWS)  and  the  controller  workstation  (CONWOR)  add  to  the 
utility  and  versatility  of  the  Janus  application  tools  available  to  the  user. 

TRAC-WSMR  has  had  an  ongoing  program  of  enhancement  and  improvement  of  Janus  since  the 
mid-1980s.  JADS  entered  into  an  agreement  with  TRAC-WSMR  (combined  in  some  cases  with 
other  ongoing  programs)  to  sponsor  various  enhancements  to  Janus  to  allow  it  to  meet  ETE  Test 
requirements. 

D2.2  Capability  to  Create,  Deploy,  and  Maneuver  Large  Numbers  of  Entities 

Janus  v6.88D  included  enhancements  to  create,  deploy,  and  maneuver  large  numbers  of  entities. 
The  number  of  units  allowed  in  Janus  was  extended  from  1200  to  6000,  then  to  8000,  and  most 
recently  to  9999  in  Janus  v6.88D.  Initial  tests  of  the  first  extension  to  6000  units  revealed  that 
version  would  not  run  at  real  time  because  of  the  updating  of  the  unit  locations  on  the  graphic 
display.  To  remedy  this  situation,  several  major  creative  changes  were  required,  and  the  design 
was  based  on  solid  computer  science  theory.  In  order  to  reduce  the  graphic  update  time  required, 
a  process  called  heterogeneous  aggregation  (HA)  was  implemented.  The  interactor  is  able  to 
d5^amically  select  the  level  of  detail  of  the  units  displayed  on  the  graphic  screen.  A  graphics 
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interface  that  allows  the  user  to  build  and  modify  a  force  according  to  military  organizations  with 
specification  to  the  entity  level  possible  at  any  level  of  aggregation  was  added  to  the  FORCE 
editor.  No  phenomenology  was  changed  for  HA  as  the  warfighting  engine  still  operates  at  the 
entity  level.  Janus  was  modified  to  allow  deployment  and  route  building  for  organizations.  The 
large  numbers  of  units  did  require  that  some  functions  not  directly  related  to  HA  be  modified. 
Even  more  changes  were  made  after  observing  users’  struggle  with  building  and  executing  large 
scenarios.  Other  model  entity  limits  that  have  been  increased  (and  their  maximum  value)  in 
Janus  v6.88D  for  a  scenario  are 

1 500  Number  of  indirect  fire  units 
400  Number  of  system  types 
100  Number  of  indirect  fire  system  types 
300  Number  of  direct  fire  system  types 
400  Number  of  direct  fire  weapon  types 
24  Number  of  missions  per  indirect  fire  unit 
60  Number  of  special  radars 
400  Number  of  actively  searching  radars  at  any  one  time 
100  Number  of  special  flyer  units 
100  Number  of  large  area  smoke  clouds 
500  Number  of  clouds 
300  Number  of  ammo  supply  units 
30  Number  of  ammo  types  for  an  ammo  supply  unit 
100  Number  of  petroleum,  oils  and  lubricants  (POL)  tankers 
500  Number  of  obstacles 
150  Number  of  nodes  in  a  route 

4000  Number  of  probability  of  hits/probability  of  kill  (PH/PK)  data  sets 


D2.3  Enhancements  to  the  DIS  Interface  in  Janus 

Prior  to  the  JADS  project,  TRAC-WSMR  used  an  external  program,  Janus  distributed  interactive 
simulation  (JanDIS),  which  ran  on  the  same  host  and  shared  memory  with  Janus,  the  combat 
model  program,  to  communicate  with  other  systems  on  a  network.  Utilization  of  JanDIS  by 
JADS  and  other  distributed  projects  revealed  that  the  arrangement  was  too  difficult  and 
cumbersome  to  use.  The  DIS  interface  was  developed  and  integrated  into  Janus  with  a  graphical 
interface  for  user  control  of  mntime  parameters.  The  PDUs  resulting  from  firing  and  detonation 
events  were  implemented  and  tested  in  the  JADS  network  in  coordination  with  the  Tactical  Army 
Fire  Support  Model. 

D2.4  Development  of  the  Janus  Plan  View  Display  (JANPVD) 

When  Janus  is  used  in  a  stand-alone  exercise,  the  JAAWS  program  is  used  as  an  after-action 
review  tool.  When  Janus  is  part  of  a  distributed  application,  information  from  entities  not  under 
control  of  Janus  would  not  be  available  to  JAAWS.  JANPVD  was  then  developed  (similar  to  the 
JAAWS  pattern)  to  display  PDUs  from  all  sources.  JANPVD  can  display  information  in  real 
time  or  it  can  run  from  recorded  data  to  provide  replay  capability.  JANPVD  displays  unit 
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positions,  losses,  direct  fires,  indirect  fires,  and  artillery  impacts.  The  user  can  select  from  six 
different  graphs  of  analysis  measures  as  a  function  of  time.  In  the  replay  mode,  the  user  has  the 
additional  capability  to  display  unit  movement  and  routes. 

D2.5  Increased  Speed  of  Line-of-Sight  (LOS)  Calculations 

Previously,  running  large  scenarios  in  Janus  resulted  in  users  having  problems  with  run  time. 
The  line-of-sight  (LOS)  calculation  in  Janus  is  the  major  consumer  of  central  processing  unit 
(CPU)  resources  when  executing  large  scenarios.  Three  software  changes  in  this  area  were 
made  to  streamline  the  LOS  process,  thus  reducing  required  CPU  resources  and  run  time. 

An  algorithm,  generally  referred  to  as  “bilateral  interpolation,”  was  used  to  determine  if  the  LOS 
ray  between  two  points  (in  three  dimensions)  was  intersected  by  the  elevation  of  the  terrain 
between  the  two  points.  While  this  algorithm  is  accurate  (in  a  theoretical  sense),  it  is  not 
operationally  efficient  since  three  interpolations  are  required  at  each  "calculation  point"  along  the 
LOS  ray.  To  remedy  this,  TRAC-WSMR  developed  and  integrated  a  new  algorithm  called 
‘"unilateral  interpolation”  for  use  in  Janus.  Except  for  the  two  endpoints,  the  new  algorithm 
requires  only  one  interpolation  at  each  calculation  point  along  the  LOS  ray.  Calculation  points 
are  at  intersections  of  the  LOS  ground-trace  with  “grid  cell”  boundaries.  Some  overhead  is 
required  to  determine  parameters  that  enable  the  fast  calculation  of  grid  cell  boundary 
intersections.  However,  only  one  interpolation  is  required  per  calculation  point  in  that  case.  In 
stand-alone  test  programs  using  representative-length  LOS  calculations,  the  new  algorithm 
appeared  to  be  about  twice  as  fast  as  the  old  algorithm. 

A  low-level  LOS  driver  routine  (subroutine  DOLOS)  performs  LOS  determination  for  both 
terrain  elevation  and  feature  data.  DOLOS  divides  a  LOS  ray  into  several  segments,  then 
considers  each  segment  one-at-a-time  to  determine  if  either  elevation  or  feature  data  would 
interfere  with  LOS  for  that  segment.  A  TRAC  LOS  study  showed  that  it  is  generally  faster 
(particularly  if  the  terrain  file  contains  a  significant  number  of  terrain  features)  to  not  divide  the 
LOS  ray  into  segments,  but  rather,  first  determine  if  elevation  data  interferes  with  LOS  (using  the 
entire  length  of  the  LOS  ray),  and  if  so,  skip  the  feature  calculations.  In  a  sense,  the  elevation 
calculation  serves  as  a  “filter”  for  the  feature  calculation.  The  speed  up  in  Janus  because  of  this 
change  was  very  difficult  to  quamtify  because  it  was  so  extremely  terrain  and  scenario  dependent. 
Suffice  it  to  say,  the  change  resulted  in  a  significant  speed  up  for  most  “typical”  Janus  scenarios. 

The  Janus  Terrain  Editor  (TED)  determines  what  types  of  terrain  features,  if  any,  are  "close"  to 
each  terrain  "grid  cell"  and  stores  that  information  in  the  terrain  file.  Janus  uses  this  information 
to  determine  what  type(s)  of  features  to  check  when  determining  if  a  particular  LOS  ray 
intersects  any  terrain  features.  The  LOS  study  showed  that  the  calculations  to  determine  possible 
intersection  of  features  with  the  LOS  ray  are  performed  much  faster  in  Janus  if  the  features  are 
sorted  by  type  so  that  all  features  of  a  given  type  are  stored  contiguously  in  memory.  Therefore, 
TED  has  been  modified  to  sort  the  terrain  features  by  type  when  the  terrain  file  is  saved.  In 
addition,  Janus  has  been  modified  to  take  advantage  of  this  fact  whenever  calculations  involving 
all  features  of  a  given  type  are  required.  Testing  indicated  that  a  significant  speed  up  for  Janus  is 
obtained  even  when  running  with  terrain  files  with  relatively  small  numbers  of  terrain  features 
(500  to  1000). 
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D3.0  Janus  Utility  Issues 

D3.1  Janus  Scenario  Development  and  Preparation 

The  Janus  scenario  preparation  process  was  quite  involved  and  composed  of  several  steps.  Two 
output  files  from  the  corps  battle  simulation  (CBS)  for  the  corps  scenario  were  provided  to 
TRAC-WSMR.  The  area  covered  for  the  corps  scenario  was  a  500  km  X  500  km  box.  The  next 
step  was  to  develop  a  Janus  map  file  that  would  contain  only  the  units  that  the  Joint 
STARSA^STARS  would  be  observing  over  the  corps  area  (the  ground  reference  coverage  area  or 
GRCA).  Units  within  the  GRCA  were  identified  from  one  of  the  CBS  files  as  were  the  number 
and  types  of  systems  within  each  unit.  If  the  number  of  units  ever  exceeded  9999,  the  units  were 
ranked  giving  targets  that  moved  during  the  scenario  priority  (i.e.,  if  targets  had  to  be  ignored 
because  of  the  9999  unit  limit,  they  would  be  stationary  targets).  The  position  location  data  of 
the  units  were  provided  on  another  file  that  had  to  be  pared  down  to  reflect  the  units  selected  in 
the  first  step.  This  task  was  facilitated  by  converting  the  file  into  an  Excel©  document  and 
sorting  it  accordingly  to  identify  what  units  to  omit.  This  completed  the  task  of  identifying  the 
organization  and  maneuver  of  the  units  for  the  Janus  scenarios.  The  force  was  deployed  in  Janus 
and  routes  for  the  moving  entities  were  provided  according  to  the  scenario  and  in  consonance 
with  the  threat  operations  plan. 

D3.2  Verification  of  Artillery  Damage  Assessment  in  Janus 

As  mention  previously,  the  information  from  the  various  intelligence  sources  represented  was 
analyzed  and  candidate  targets  were  selected  for  the  ATACMS.  The  candidate  targets  were  sent 
to  the  D&SA  BL  where  it  was  determined  if  an  ATACMS  would  be  fired  at  one  or  more  of  the 
candidate  targets.  Information  describing  the  missile  firings  was  then  sent  back  to  Janus  for 
assessment  of  the  damage.  The  D&SA  BL  noted  that  the  resulting  damage  assessment  for  the 
ATACMS  from  Janus  appeared  to  be  lower  than  what  they  thought  it  should  be.  The  Janus 
algorithms  for  ATACMS  representation  and  damage  assessment,  together  with  supporting  data, 
were  extensively  reviewed  and  checked  out  to  ensure  conformity  with  joint  munitions 
effectiveness  manual  (JMEM)  methodology  and  Army  material  systems  analysis  activity 
(AMSAA)  data. 

D4.0  Conclusions 

The  enhancements  made  to  Janus  to  meet  the  JADS  requirements  are  of  broad  general  use  to  all , 
Janus  users  and  have  resulted  in  a  tremendous  benefit  to  the  U.S.  Army.  Of  the  many  benefits  of 
the  JADS  project,  the  addition  of  these  capabilities  to  the  Janus  model  has  certainly  been 
significant. 
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Appendix  E  -  EXE  Test  Program  Work  Breakdown  Structure 
El.O  Background 

JADS  personnel  worked  with  the  MURE  Corporation  to  produce  an  advanced  distributed 
simulation  (ADS)  cost  model  guidance  (CMG)  document.  To  develop  the  CMG,  MITRE  created 
a  work  breakdown  structure  (WBS)  relative  to  ADS-based  testing.  (For  the  latest  version  of  the 
ADS  WBS,  please  contact  the  JADS  Joint  Test  Force.)  Section  3  identifies  the  program  costs  for 
the  End-to-End  (ETE)  Test,  as  applied  against  the  most  recent  version  of  the  ADS  WBS. 

E2.0  WBS  Terms  and  Definitions 

The  following  terms  and  definitions  were  used  in  the  ETE  Test  WBS. 

1.x  System  Test  and  Evaluation  (T&E).  This  element  refers  to  the  use  of  prototype, 
production,  or  specifically  fabricated  hardware  and  software  to  obtain  or  validate  engineering 
data  on  the  performance  of  the  prime  mission  equipment  (PME).  This  element  includes  the 
detailed  planning,  execution,  support,  data  reduction,  and  reports  from  such  testing  (exclusive  of 
those  required  under  data),  and  all  hardware  items  which  are  consumed  or  planned  to  be 
consumed  in  the  conduct  of  such  testing.  It  includes  all  effort  associated  with  the  design  and 
production  of  models,  specimens,  fixtures,  and  instrumentation  in  support  of  the  system-level  test 
program.  It  also  includes  all  planning,  management,  and  coordination  of  federation  developers 
required  to  incorporate  ADS  use  into  traditional  T&E  methods. 

1.X.0  FeasibCity  Analysis.  This  element  provides  the  decision  maker  (usually  a  program 
manager)  with  the  necessary  information  based  on  quantitative  and  qualitative  analyses  on  which 
a  decision  can  be  made  to  use  or  not  use  ADS  in  the  program’s  T&E  phase.  Also,  if  ADS  will  be 
used,  this  analysis  should  support  identification  of  how  it  can  best  be  used  to  supplement 
traditional  T&E  capabilities  and  what  resources  may  be  available  and  their  location. 

l.X.0.1  Test  Planning  Methodology.  An  approach  to  analyzing  applicability  of  ADS  in 
meeting  a  program’s  test  objectives.  Included  in  this  approach  are  a  survey  of  availability  of 
models  and  simulations,  networks,  facilities,  trained  personnel,  etc.,  to  support  a  program’s  T&E 
requirements.  Also,  this  analysis  addresses  the  program’s  required  security  and  the  ability  and 
availability  of  equipment  and  personnel  who  can  sustain  the  program’s  security  integrity. 

l.X.0.2  Rough  Order  of  Magnitude  (ROM)  Cost  Analysis.  Perform  a  ROM  cost  estimate 
analyzing  cost  of  applying  ADS.  Costs  related  to  development,  modification,  and  /or 
procurement  of  modeling  and  simulation  (M&S),  software,  hardware,  networks,  facilities, 
personnel,  training,  validation,  verification  and  accreditation  (W&A),  and  documentation  are 
some  examples  of  elements  that  should  be  included  in  the  estimate. 

l.X.0.3  Risk  Analysis.  Perform  an  initial  analysis  of  technical  and  programmatic  risks  of 
incorporating  or  not  incorporating  ADS  methods  into  the  traditional  T&E  approach. 
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If  program  management  has  reached  a  decision  to  incorporate  ADS  into  the  T&E  phase,  some 
WBS  elements  will  need  to  be  modified,  while  others  will  have  to  be  added.  The  following 
WBS  elements  include  tasks  required  to  incorporate  ADS  into  a  program’s  developmental  test 
and  evaluation  (DT&E)  phase.  These  WBS  elements  are  not  stand-alone  products  and  are  not 
representative  of  all  activities  necessary  for  a  successful  T&E  phase. 

1.x.  1  Development  Test  And  Evaluation  (DT&E).  Test  and  evaluation  is  conducted  on 
systems  to  (a)  demonstrate  that  the  engineering  design  and  development  process  is  complete,  (b) 
demonstrate  that  the  design  risks  have  been  minimized,  (c)  demonstrate  that  the  systems  will 
meet  specifications,  (d)  estimate  the  system's  military  utility  when  introduced,  (e)  determine 
whether  the  engineering  design  is  supportable  (practicable,  maintainable,  and  safe)  for 
operational  use,  (f)  provide  test  data  with  which  to  examine  and  evaluate  trade-offs  against 
specification  requirements,  life  cycle  cost,  and  schedule,  and  (g)  perform  the  logistics  testing 
efforts  to  evaluate  the  achievement  of  supportability  goals,  and  the  adequacy  of  the  support 
package  for  the  system,  e.g.,  deliverable  maintenance  tools,  test  equipment,  technical 
publications,  maintenance  instructions,  and  personnel  skills  and  training  requirements,  etc. 
DT&E  is  planned,  conducted  and  monitored  by  the  developing  agency  of  the  Department  of 
Defense  (DoD)  component. 

DT&E  tasks  are  the  planning,  conducting,  and  reporting  of  tests.  Planning  tasks  include 
selecting  the  tools  such  as  the  facilities,  equipment,  personnel,  software,  etc.,  needed  to  conduct 
the  tests,  determining  the  sequence  of  tests  to  be  performed,  writing  detailed  test  procedures  for 
individual  tests,  and  determining  the  expected  test  outputs.  Also,  before  the  tests  are  conducted, 
the  necessary  hardware  and  software  integration  are  done.  Then  the  tests  are  conducted.  Test 
environments  progress  from  least  stressful  to  the  most  stressful.  The  test  program  encompasses 
system  integration  and  performance  tests  as  well  as  reliability,  maintainability,  and 
environmental  tests.  The  PME,  to  a  large  degree,  determines  specific  DT&E  tests.  After  the 
tests  are  conducted,  the  collected  data  are  reduced  and  analyzed.  Data  reduction  is  the  filtering 
and  statistical  assessment  of  the  raw  data  for  specific  characteristics  and  information.  The  result 
of  the  test  analysis  is  the  test  report. 

1.x.  1.1  Planning.  The  purpose  of  this  section  is  to  initiate  the  studies  and  analyses  of  ADS 
requirements,  architecture,  resources,  and  constraints,  define  objectives,  and  identify  critical 
systems.  As  part  of  the  planning  effort,  organizations  that  will  be  used  in  the  simulation 
federation  should  be  contacted  arid  talks  initiated  toward  achieving  the  required  agreements  on 
responsibilities  and  schedules.  Note  that  several  of  these  activities  will  continue  definitional 
work  started  in  1  .X.0. 1 . 

l.X.  1.1.1  ADS  Requirements  Definition.  Define  ADS  test  objectives,  measures  of 
effectiveness  and  performance,  needed  resources,  and  constraints.  Initiate  development  of  plans 
and  procedures  for  directing  architecture  and  test  toward  meeting  objectives. 

1.x.  1.1. 1.1  Requirements  Identification.  Develop  an  initial  problem  statement  from 

information  available  at  this  stage  of  development. 
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1.x.  1.1. 1.1.1  Critical  Systems  of  Interest  Description.  Develop  a  clear  and  unambiguous 
statement  of  program  needs  to  include  a  high-level  description  of  critical  systems  of  interest, 
required  fidelity  and  resolution  of  simulated  entities,  and  output  data  requirements. 

l.X.1.1.1.1.2  Support  Resources  Availability.  Identify  the  resources  that  will  be  available  to 
support  the  ADS  architecture  (funding,  personnel,  tools,  facilities,  etc.). 

l.X.1.1.1.1.3  Programmatic  Risk  Identification.  Determine  known  constraints  that  could 
affect  the  ADS  architecture  development,  e.g.,  due  dates,  security  requirements,  etc. 

1.x.  1.1. 1.2  Objectives  Development.  Refine  the  statement  of  program  needs  into  a  more 
detailed  set  of  specific  objectives  and  plans  for  the  ADS  architecture. 

l.X.  1.1. 1.2.1  Test  Objectives,  Scenarios,  Conditions  and  Measures  Identification.  Develop 
a  prioritized  list  of  measurable  objectives  for  the  ADS  architecture.  Assess  availability  of 
government  or  contractor  furnished  equipment,  facilities  and  data.  Define  operational  context 
constraints  or  preferences  including  geographical  regions,  environmental  conditions,  threats  and 
tactics. 

l.X.  1.1. 1.2.2  Plan  Development.  Develop  the  ADS  architecture  and  network  implementation 
plans  showing  planned  schedule  and  major  milestones.  Develop  the  ADS  configuration 
management  plan,  data  collection  test  plan,  and  initial  verification,  validation  and  accreditation 
(VV&A)  test  plan. 

1.x.  1.1. 1.2.3  Security  Requirements  Identification.  Identify  security  position  including 
expected  security  level  and  possible  designated  approval  authority. 

l.X.  1.1. 1.2.4  Support  Tools  Selection.  Identify  and  select  tools  to  support  scenario 
development,  concept  analysis,  configuration  management,  VV&A  and  test  activities.  Tool 
selection  should  be  based  on  tool  availability,  cost,  maintainability,  applicability  to  the  given 
function,  and  the  ability  of  a  given  set  of  tools  to  exchange  data  within  the  ADS  architecture. 

1.x.  1.1. 2  Establish  Federate  Development  Team.  Designate  members  of  federating  system’s 
organizations  to  coordinate  development  of  ADS  environment  including  the  architecture, 
federation  object  model  (FOM),  schedules,  and  other  test  requirements.  Define  requirements, 
expectations,  test  objectives,  etc.,  to  provide  these  organizations  with  an  understanding  of  what 
resources  are  expected  from  them  in  personnel,  training,  equipment,  and  what  costs  they  may  be 
required  to  bear. 

l.X.  1.1. 3  Reporting  And  Documentation.  Document  and/or  report  on  all  plans  and  procedures 
required  achieving  the  objectives  of  the  elements  associated  with  l.X.  1  to  include  all  technical, 
management,  programmatic  reports,  plans,  briefings,  engineering  drawings,  etc. 
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1.x.  1.2  Concept  Development  The  purpose  of  this  section  is  to  develop  a  representation  of  the 
real-world  domain  of  interest  (entities  and  tasks)  in  terms  of  a  set  of  required  objects  and 
interactions. 

1.x.  1.2.1  Scenario  Development.  Develop  a  functional  specification  of  the  ADS  test  scenario, 
using  the  operational  context  constraints  identified  in  l.X.1.1.1.2. 

1.x.  1.2. 1.1  Entities  Identification.  Identify  all  entities  that  must  be  represented  in  simulations 
or  otherwise  by  the  ADS  architecture. 

1.x.  1.2. 1.2  Functional  Description  of  Entities.  Document  functional  descriptions  of  the 
capabilities,  behavior,  and  relationships  among  entities  over  time. 

1.x.  1.2. 1.3  Relevant  Environmental  Conditions.  Identify  relevant  environmental  conditions 
that  impact  or  are  impacted  by  entities  in  the  ADS  architecture  including  initial  and  terminal 
conditions  (e.g.,  textual  scenario  descriptions,  event-trace  diagrams,  and  graphical  illustrations  of 
force  laydowns  and  communication  paths). 

IX.  1.2. 1.4  Conceptual  Model  Development.  Select  technique  for  developing  the  conceptual 
model  (e.g.,  static  process  flow  diagram,  process  flow  diagrams,  correlation  tables  of  objects  and 
activities,  descriptive  texts)  and  initiate  development  to  capture  all  features  of  the  test 
environment. 

l.X.  1.2.2  Concept  Analysis.  Develop  an  implementation-independent  representation  of  ADS 
architecture. 

l.X.  1.2.2. 1  Entity  Characteristics,  Relationships  and  Interactions  Evaluation.  Identify  all 
entity  characteristics  and  the  static  and  dynamic  relationships  among  them.  Identify  behavioral 
and  transformational  (algorithmic)  aspects  of  each  using  the  ADS  architecture  scenario. 

I.X.1. 2.2.2  Entities  Representation.  Determine  entity  characteristics  (attributes)  and 
interaction  descriptors  (parameters). 

1.x.  1. 2.2.3  ADS  Architecture  Requirements  Determination.  Determine  ADS  architecture 
requirements  such  as  data,  latency,  synchronization,  network,  interface,  test  control  and 
monitoring. 

1.x.  1.2.3  Reporting  And  Documentation.  Develop  all  plans,  procedures,  reports  and 
documentation  required  to  meet  the  objectives  of  the  elements  associated  with  l.X.  1.2. 

1.x.  1.3  Design  And  Development.  The  purpose  of  this  section  is  to  do  detailed  design  of  the 
test  environment  to  meet  the  test  objectives  and  ensure  adequate  data  collection;  procure  the 
hardware  and  software,  and  finalize  schedules  for  installation,  integration,  VV&A,  and  test 
execution. 
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1.x.  1.3.1  ADS  Architecture  Design.  Design  the  ADS  architecture  that  will  be  required  to  test 
and  evaluate  the  system. 

l.X.1.3.1.1  ADS  Architecture  Detailed  Design.  Design  a  systems  engineering  methodology  to 
support  ADS  architecture  development  and  integration.  This  requires  close  coordination  among 
all  facilities  and  entity  participants  to  ensure  a  common  understanding  of  the  ADS  architecture 
goals  and  requirements. 

l.X.  1.3. 1.1.1  Node  Site  Surveys.  Select  ADS  nodes  and  survey  each  location.  Nodeselection 
is  based  on  fidelity,  availability,  cost  and  schedule.  Site  surveys  determine  facility 
communications  architecture  and  requirements  and  space  requirements  for  tester-supplied 
equipment  and  personnel. 

l.X.  1.3. 1.1. 2  Test  Control  And  Analysis  Center  (TCAC)  Definition.  Define  required  TCAC 
capabilities  and  optimal  location. 

l.X.  1.3. 1.1. 3  Security  Approach  Development.  Perform  security  risk  assessment  and  develop 
a  security  concept  of  operations. 

l.X.  1.3. 1.1. 4  ADS  WAN  Requirements  DeHnition.  Define  ADS  Wide  Area  Network  (WAN) 
requirements  (bandwidth,  data  rate,  acceptable  latency,  network  management/control 
responsibility). 

l.X.  1.3. 1.1. 5  ADS  Network  Components  Determination.  Determine  if  DoD-sponsored 
networks  meet  the  ADS  WAN  requirements  or  if  commercial  resources  are  required.  If  DoD- 
sponsored  common-user  services  are  not  viable,  application  for  waiver  from  the  Defense 
Information  Systems  Agency  (DIS  A)  and  contract  for  leased  lines  would  be  required. 

l.X.1.3.1.1. 6  ADS  Network  Hardware  Selection.  Select  hardware  for  the  ADS  network 
(routers,  channel  service  units,  data  service  imits,  multiplexers,  encryptors). 

l.X.  1.3. 1.1. 7  Test  Control  Hardware  and  Software  Selection.  Select  test  control  hardware 
and  software  based  on  the  data  and  voice  communications  requirements  and  TCAC  space 
requirements  and  layout. 

l.X.  1.3. 1.1. 8  Data  Collection  and  Instrumentation  Requirements  Determination.  Select 
instramentation  types  and  data  logging  software  based  on  data  requirements.  Determine 
hardware  requirements  for  data  storage  and  handling  (tape  drivers,  compact  disks,  optical  disks). 
Develop  handling  procedures  for  collecting  and  storing  data  from  the  distributed  network. 

l.X.  1.3. 1.1. 9  Time  Synchronization  Method  Determination.  Determine  method  for 

synchronizing  time  stamps  for  data  loggers.  Select  synchronization  hardware  and  software. 

l.X.1.3.1.1.10  ADS  Architecture  Development  And  Integration  Plan.  Complete  final  version 
of  formalized  plan  for  ADS  architecture  development  and  integration. 
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1.x.  1.3. 2  ADS  Architecture  Development.  Initiate  procurement  and/or  development  of  tools 
and  identification  of  other  procedures  to  build  the  ADS  architecture. 

l.X.l. 3.2.1  ADS  Network  Procurement.  Procure  or  develop  ADS-specific  hardware  tools  for 
network  analysis  and  monitoring. 

I.X.1. 3.2.2  Secure/Encrypted  Operations  Procedure  Development  and  Approval.  Develop 
secure/encrypted  operations  procedures  for  approval  by  the  designated  approval  authority  (DAA). 
Achieve  formal  security  accreditation  for  all  network  and  facilities. 

l.X.l. 3.2.3  Hardware/Software  Configuration  Control  Implementation.  Implement  strict 
hardware/software  configuration  control. 

1.x.  1.3. 2.4  Data  Protocols  Development.  Develop  standard  data  protocols  to  satisfy  ADS 
architecture  communications  and  interoperability  requirements. 

I.X.1. 3.2.5  Interface  Design,  Build,  or  Procurement.  Design,  build,  or  procure  simulation 
and  network  interfaces,  runtime  infrastructure  (RTI)  interfaces  (if  required),  and  special  purpose 
interfaces. 

I.X.1. 3.2.6  Simulation  Modifications.  Identify  and  implement  modifications  necessary  to  use 
external  inputs  and  to  generate  required  outputs  from  existing  simulations. 

I.X.1. 3.2.7  Range  Data  Processing  Modifications.  Identify  and  implement  modifications 
necessary  to  meet  time-space-position-information  accuracy,  smoothness,  and  latency 
requirements. 

I.X.1. 3.2.8  Facilities  Modifications.  Identify  and  implement  modifications  required  for  replay 
capability  to  be  used  during  integration  testing. 

l.X.l. 3.3  Reporting  and  Documentation.  This  element  includes  ADS-specific  data. 

This  element  includes,  for  example,  all  plans,  procedures,  reports  and  documentation 
required  to  meet  the  objectives  of  the  elements  associated  with  l.X.l. 3. 

1.x.  1.4  Installation,  Integration  and  Test  The  purpose  of  this  section  is  to  install  and 
integrate  the  ADS  hardware  and  software  subcomponents  and  to  conduct  testing  to  ensure  that 
the  ADS  architecture  meets  interoperability  requirements. 

l.X.l. 4.1  Execution  Planning.  Define  requirements  to  support  execution  of  the  ADS 
architecture. 

1.x.  1.4. 1.1  Integration  Test  Plan.  Develop  integration  test  plan  to  support  incremental  checks 
on  configuration  during  ADS  architecture  build-up. 
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1.x,  1.4. 1.2  Test  Control  Procedures.  Develop  test  control  procedures  for  ADS  architecture 
integration  and  test. 

l.X.1.4.1.3  Detailed  Test  Execution  Plan.  Develop  detailed  test  execution  plan  for  ADS 
architecture  integration  and  test. 

1.x.  1.4. 1.4  Security  Test  and  Evaluation  Plan.  Develop  security  test  and  evaluation  plan  that 
defines  procedures  for  testing  and  evaluating  network  security. 

1.x.  1.4.2  ADS  Architecture  Integration  and  Test  Execution.  Integrate  all  participating 
facilities  and  entities  into  one  operating  environment  and  verify  that  all  components  are 
interoperable  to  the  degree  required  to  achieve  the  ADS  architecture  objectives. 

1.x.  1.4.2.  Hardware/Software  Installation.  Install  and  check  out  network  hardware  and 
software. 

l.X.l. 4.2.2  Compliance  Testing.  Perform  individual  compliance  testing  for  each  facility  and 
node  to  verify  correct  implementation  of  ADS  architecture  requirements. 

1.x.  1. 4.2.3  Integration  Testing.  Perform  incremental  integration  testing,  using  iterative  “test- 
fix-test”  approach  including  replay  of  trials  to  diagnose  problems  and  verify  fixes.  This  includes 
a)  check-out  of  interfaces  and  facility  modifications  with  linking  between  pairs  of  nodes,  b) 
baseline  performance  of  network  with  no  loading  from  the  simulations  or  entities,  c)  test 
performance  of  critical  portions  of  network  under  loading  representative  of  test  conditions. 

l.X.l. 4.2.4  Risk  Reduction  Missions.  Execute  scenarios  with  fully  linked  test  execution 
configuration.  Include  security  certification  (if  required)  and  evaluation  of  test  control  and 
monitoring  procedures  as  well  as  data  collection  and  analysis  procedures. 

l.X.l. 4.3  Verification,  Validation  and  Accreditation  (VV&A).  Obtain  VV&A  of  the  ADS 
architecture.  Consider  VV&A  policies  and  procedures  for  each  individual  entity  as  well  as 
system  compliance,  compatibility  and  interoperability  requirements  for  the  ADS  architecture. 

l.X.l. 4.3.1  ADS  Verification.  Verify  that  the  ADS  architecture  accurately  represents  the 
developer’s  concept  description  and  specifications  and  meets  the  needs  stated  in  the  requirements 
document. 

l.X.l. 4.3. 1.1  Distributed  Network  Verification.  Verify  that  the  distributed  network  meets  the 
ADS  architecture  requirements. 

l.X.l. 4.3. 1.2  Integration  Testing.  Test  to  determine  that  the  integrated  environment  operates 
as  required  and  meets  specifications. 
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LX.  1. 4.3.2  ADS  Validation.  Determine  the  extent  to  which  the  ADS  architecture  represents 
the  real  world  from  the  perspective  of  its  intended  use,  i.e.,  it  meets  requirements  of  test 
objectives. 

1.x.  1.4.3. 2.1  Test  Environment  Validation.  Validate  the  test  environment  to  the  VV&A  plan 
specifications. 

1.x.  1.4.3. 2.2  Comparison  Standard  Validation.  Validate  the  ADS  architecture’s  performance 
against  the  comparison  standard. 

1.x.  1. 4.3.3  Accreditation.  Obtain  an  official  determination  (based  on  experience  and  expert 
judgement)  that  the  ADS  architecture  is  acceptable  for  its  intended  purpose. 

1.x.  1.4.3. 3.1  Network  Accreditation  Requirements.  Define  the  network  accreditation 
requirements. 

l.X.1.4.3.3.2  Synthetic  Environment 

1.x.  1.4.4  Reporting  and  Documentation.  This  element  includes  ADS-specific 
This  element  includes,  for  example,  all  plans,  procedures,  reports  and  documentation 
required  to  meet  the  objectives  of  the  elements  associated  with  l.X.1.4. 

LX.  1.5  Test  Execution  and  Analysis.  The  purpose  of  this  section  is  to  conduct  the  tests, 
collect  data,  analyze  test  results,  and  provide  feedback  to  the  program  manager  and  developer. 

LX.  1.5.1  Test  Readiness  Review.  Conduct  a  review  to  assess  ADS  architecture  readiness  for 
formal  testing. 


LX.  1.5.2  Test  Execution.  Exercise  all  components  of  the  ADS  architecture  as  an  integrated 
whole  to  achieve  the  test  objectives. 

LX.  1.5.2. 1  Data  Collection.  Collect  data  in  accordance  with  approved  data  collection 
procedures. 

LX.  1.5.3  Results  and  Feedback.  Analyze  test  execution  data  and  provide  report  to  the 
program  manager  and  developer. 

LX.  1.5.3 . 1  Data  Analysis.  Analyze  outputs  from  the  test  execution  phase. 

LX.  1.5.3. 2  Test  Evaluation.  Evaluate  results  from  the  execution  phase  to  determine  if  all  test 
objectives  have  been  met.  If  all  objectives  were  not  met,  identify  corrective  actions  to  implement 
for  retest. 

LX.  1.5.3. 3  Provide  Test  Report.  Deliver  test  report  and  legacy  products  to  customer  and  data 
repository. 
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1.X.2  Operational  Test  And  Evaluation  (OT&E).  This  element  addresses  test  and  evaluation 
conducted  by  agencies  other  than  the  developing  command  to  assess  the  prospective  systems 
military  utility,  operational  effectiveness,  operational  suitability,  logistics  supportability 
(including  compatibility,  interoperability,  reliability,  maintainability,  and  logistics  requirements), 
cost  of  ownership,  and  need  for  any  modifications.  The  initial  operational  testing  and  evaluation 
(lOT&E)  conducted  during  the  development  of  a  weapon  system  is  included  in  this  element. 
This  element  encompasses  such  tests  as  integrated  system  tests,  flight  tests  and  sea  trials, 
mobility  demonstrations,  on-orbit  tests,  spin  demonstrations,  stability  tests,  etc.,  and  support 
thereto,  required  to  prove  the  operational  capability  of  the  deliverable  system.  Contractor 
support  (e.g.,  technical  assistance,  maintenance,  and  labor  material,  etc.)  consumed  during  this 
phase  of  testing  is  also  included  in  this  element. 

If  ADS  was  used  during  T&E,  the  ADS  architecture  can  be  reused.  If  modifications  are  required, 
some  of  the  planning  and  preparation  elements  must  be  revisited  prior  to  executing  the  test.  If 
ADS  was  not  used  for  developmental  test  and  evaluation  (DT&E),  all  elements  from  l.X.1.1 
through  l.X.l. 5.3.3  must  be  incorporated  into  the  OT«feE  process. 

E3.0  EXE  Test  Work  Breakdown  Structure  (WBS) 

ETE  Test  Overall  Observations: 

Ideally,  the  WBS  data  collection  process  should  be  in  place  at  the  start  of  the  ADS  test.  This 
process  will  include  the  development  and  maintenance  of  a  dictionary  of  WBS  terms  and  the 
creation  of  a  manual  describing  the  proper  application  of  the  WBS  terms  to  ADS  test  activities. 
In  addition,  the  process  requires  a  point  of  contact  for  questions  with  respect  to  implementing 
the  process  accessible  by  ADS  test  members  throughout  the  life  of  the  ADS  test. 

Valid  WBS  data  must  be  collected  from  all  contractors  and  government  organizations  supporting 
the  ADS  test.  The  contractors  and  their  subcontractors  can  be  tasked  to  provide  WBS  data 
through  contract  deliverables.  The  government  organizations  can  be  asked  to  provide  WBS  data 
via  specific  provisions  in  memoranda  of  agreements  between  the  ADS  test  team  and  the 
government  organizations. 

1.x  System  Test  and  Evaluation 

1.X.0  Feasibility  Analysis 

During  1993-1994,  a  joint  feasibility  study  (JFS)  was  conducted  to  determine  the  necessity  and 
feasibility  of  a  joint  test  and  evaluation  (JT&E)  of  distributed  interactive  simulation.  This  study 
defined  the  JADS  System  Integration  Test,  End-To-End  Test,  and  Electronic  Warfare  Test. 

The  JFS  took  13  months  and  cost  $523,000;  $460,500  for  contractor  support  while  $62,500  was 
for  government  travel.  Each  service  provided  additional  funding  for  travel.  The  Air  Force 
Operational  Test  and  Evaluation  Center  (AFOTEC)  funded  and  provided  day-to-day  operating 
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costs,  office  equipment,  and  work  areas.  The  JADS  JFS  team  included  24  government  and 
contractor  personnel. 


“JADS  JT&E  Joint  Advanced  Distributed  Simulation 
Joint  Feasibility  Study  Final  Report.  ”  February  1995. 

ETE  Test  Observation: 

The  future  success  of  an  ADS  test  is  very  much  dependent  on  the  level  of  rigor  associated  with  a 
preliminary  feasibility  analysis.  Having  the  right  personnel  with  the  necessary  backgrounds  and 
being  able  to  do  the  necessary  research,  the  JADS  JFS  was  able  to  thoroughly  and  accurately 
define  the  ETE  Test  in  terms  of  projected  schedule,  cost,  and  manpower  requirements.  Since 
these  initial  forecasts  were  realistic,  the  ensuing  ETE  Test  could  be  successfully  completed 
without  any  disruptions  because  of  funding  or  personnel  shortages  or  lack  of  time.  In  addition, 
during  the  course  of  the  test,  the  ETE  Test  management  could  concentrate  on  technical  and 
programmatic  issues  and  not  devote  their  energies  to  solving  significant  financial,  personnel,  or 
scheduling  problems. 

1.X.1  Development  Test  and  Evaluation  (DT&E).  No  WBS  data  available. 

Northrop  Grumman  is  currently  developing  an  annual  release  of  the  radar  software  that  will 
incorporate  a  revised  version  of  the  A-tracker  software.  This  provided  JADS  with  the 
opportunity  to  ask  Northrop  Grumman  to  conduct  an  ad  hoc  study,  parallel  to  the  normal  testing 
of  the  A-tracker  software,  to  determine  if  it  would  be  possible  to  use  the  Virtual  Surveillance 
Target  Attack  Radar  System  (VSTARS)  to  test  this  software. 

Numerous  problems  were  experienced  because  of  the  ad  hoc  nature  of  the  study,  both  in  the  areas 
of  software  integration  and  scenario  generation.  Additionally,  the  verification  and  validation  of 
VSTARS  had  not  yet  been  completed  and  thus  no  results  could  be  used  as  documentation  for  the 
annual  release.  Despite  these  problems,  several  lessons  learned  resulted  from  this  study. 

l.X.2.  Operational  Test  and  Evaluation  (OT&E) 

NOTES: 

1 .  OT&E  encompasses  Phases  1-4  of  the  ETE  Test. 

2.  ETE  Test  team  expenses  during  Phases  1-4  were  as  follows: 


Phase  1 

Phase  2 

Phase  3 

Phase  4 

Hardware  and 

Software  Purchases 

$21,040 

$26,719 

$  925 

$ 

0 

Travel 

$85,530 

$69,700 

$39,650 

$8,900 

Training 

$  7,790 

$  3,889 

$  0 

_  $ 

0 

Shipping 

$  0 

$  2,578 

$  82 

$ 

0 
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ETE  Test  Observations: 


Test  team  travel  costs  may  be  significant  during  all  phases  of  an  ADS  test. 

Allocate  funds  in  the  early  stages  of  an  ADS  test  for  test  team  training. 

l.X.2.1.  Planning 

l.X.2.1.1  ADS  Requirements  Identification 

Appendix  I  {Program  Test  Design )  of  the  JADS  Joint  Feasibility  Study  ( February  1995)  provided 
the  initial  identification  of  ADS  requirements  with  respect  to  the  End-To-End  Test.  According  to 
Figure  1-1,  “JADS  JFS  Schedule,”  of  the  JADS  Joint  Feasibility  Study,  the  program  test  design 
took  about  six  months  to  develop. 

The  JADS  Joint  Test  and  Evaluation  Program  Test  Plan  (February  1996)  provided  a  more 
detailed  discussion  of  ADS  requirements  with  respect  to  the  End-To-End  Test. 

l.X.2.1.2  Establish  Federate  Development  Team.  Not  applicable  to  ETE  Test. 

l.X.2.2.  Concept  Development 

l.X.2.2.1  Scenario  Development 

Janus  6.88D  development . $852,000 

ETE  Test  Observation: 

Contracted  scenario  development  can  be  expensive.  In  the  case  of  the  End-To-End  Test, 
scenario  development  costs  were  about  11%  of  total  test  costs. 

l.X.2.2.2  Concept  Analysis 

The  JADS  Joint  Test  and  Evaluation  Program  Test  Plan  (February  1996)  provided  a 
discussion  of  concept  analysis  with  respect  to  the  End-To-End  Test. 

l.X.2.3  Design  and  Development 

l.X.2.3.1  ADS  Architecture  Design 

l.X.2.3.1.1.1  Node  Site  Surveys . .  1  day/node 

l.X.2.3.2  ADS  Architecture  Development 
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l.X.2.3.2.1  ADS  Network  Procurement 


Network  equipment  and  instrumentation . $173,720 

Network  leases  (Phases  1-4) . $379,488 

Joint  Surveillance  Target  Attack  Radar  (STARS)  Joint  Task  Force . $301,476 

Fort  Hood  (Test  and  Experimentation  Command)  node . $239,465 

Fort  Sill  node . $101,091 


ETE  Test  Observations: 

The  sophisticated  networking  equipment  and  instrumentation  required  for  an  ADS  test  can  be 
expensive. 

ADS  testers  need  to  budget  for  recurring  costs  (i.e.,  network  lease  payments)  as  well  as  initial 
costs. 

The  support  provided  by  other  facilities  to  an  ADS  test  can  also  be  expensive. 

l.X.2.3.2.5  Interface  Design,  Build,  or  Procurement 


Surveillance  control  data  link  (SCDL)  development . $327,497 

l.X.2.3.2.6  Simulation  Modifications 

VSTARS  development  (including  contracting  fee) . $4,372,413 

Advanced  Radar  Imaging  Emulation  System  (ARIES) 
development  (including  contracting  fee) . $901,474 


ETE  Test  Observation: 

Contracted  simulation  development  and  modification  can  be  expensive. 

l.X.2.3.3  Reporting  and  Documentation 

Phase  1  ETE  Test  report . 7  months 

l.X.2.4  Installation,  Integration,  and  Test 
l.X.2.4.1  Execution  Planning 

l.X.2.4.1.2  Test  Control  Procedures . 1  month 
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l.X.2.4.1.3  Detailed  Test  Execution  Plan 


Phase  1  test  activity  plan  development,  revision,  and  coordination . 8  months 

Phase  2  test  activity  plan  development,  revision,  and  coordination 

(2  individuals  worked  full  time) . 4  months 

Phase  3  test  activity  plan  development,  revision,  and  coordination 

(2  individuals  worked  full  time) . 3  months 

Phase  4  test  activity  plan  development,  revision,  and  coordination 
(1  individual  worked  full  time) . 2  months 


ETE  Test  Observation: 

A5  an  ADS  test  moves  through  its  various  phases,  test  team  personnel  will  become  more 
proficient  at  writing  the  necessary  test  execution  plans,  resulting  in  successively  lower  test  plan 
production  costs, 

l.X.2.4.2  ADS  Architecture  Integration  and  Test  Execution 


l.X.2.4.2.1  Hardware/Software  Installation... - about  1  week/node 

l.X.2.4.2.3  Integration  Testing 

Phase  2  ETE  Test:  functionality  and  integration  tests  #1-4 . 19  days 

Phase  3  ETE  Test:  25-26  Feb  99  and  10-12  Mar  99  testing . 5  days 

l.X.2.4.2.4  Risk  Reduction  Missions 

Phase  2  ETE  Test:  pretest  data  reduction  and  analysis  rehearsals . 60  hours 

Phase  2  ETE  Test:  risk  reduction  tests . 5  days 

Phase  4  ETE  Test:  15-17  Mar,  24  Mar,  and  29-30  Mar  99  testing . 6  days 


ETE  Test  Observation: 

The  costs  of  risk  reduction  testing  are  outweighed  by  their  effect  of  improving  the  probability  of 
success  of  the  actual  ADS  testing. 
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l.X.2.4.3  Validation,  Verification,  and  Accreditation  (VV&A) 
l.X.1.4.3.1  ADS  Verification 
l.X.1.4.3.2  ADS  Validation 

Prepare,  coordinate,  and  revise  Verification  and  Validation  Plan 
for  the  End-to-End  (ETE)  Test  (Appendix  C  to  Version  2.0  of  the 
JADS  Joint  Test  Force  End-To-End  Test  Activity  Plan,  September  1998 


(1  individual  assigned  to  this  task) . 1  month 

Prepare,  coordinate,  and  revise  Phase  2  verification  and  validation 
results  for  the  End-To-End  Test 

(1  individual  assigned  to  this  task) . 1  month 

Phase  3  ETE  Test:  13  Mar  99  testing . . . 1  day 

l.X.1.4.3.3  Accreditation 

Prepare  and  convene  accreditation  board  on  3  Feb  98 . 2  days 

Prepare  and  convene  accreditation  board  on  1  Sep  98 . 2  days 

Prepare  and  convene  accreditation  board  on  17  Feb  99 . 2  days 

•  Prepare  for  accreditation  board  planned  for  Jun  99 . 1  day 

l.X.2.4.4  Reporting  and  Documentation 

Phase  1  ETE  Test  report  development,  revision,  and  coordination . 7  months 


l.X.2.5  Test  Execution  and  Analysis 


NOTES: 

1.  Phase  2  test:  encompasses  14  Sep  98-6  Oct  98  testing 

2.  Phase  3  test:  encompasses  13  Mar  99  testing 

3.  Phase  4  test:  encompasses  19  Mar,  25  Mar,  and  31  Mar  99  testing 

l.X.2.5.1  Test  Readiness  Review 

During  ETE  Test  Phases  1-4:  At  least  nine  test  readiness  reviews 
and  four  integrated  product  team  meetings . 21  meeting  days 

ETE  Test  Observation: 

The  costs  of  test  readiness  reviews  and  integrated  product  team  meetings  are  outweighed  by 
their  effect  of  improving  communication  within  the  ADS  test  team,  therefore  increasing  the 
probability  of  success  of  actual  ADS  testing. 
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l.X.2.5.2  Test  Execution 


l.X.2.5.2.1  Data  Collection . about  1  hour/test  trial 

ETE  Test  Observation: 

An  ADS  test  can  enjoy  greatly  reduced  labor  costs  associated  with  the  data  collection  process. 

l.X.2.5.3  Results  and  Feedback 

l.X.2.5.3.1  Data  Analysis . about  8-10  hours/test  trial 

l.X.2.5.3.2  Test  Evaluation . about  1-2  hours/test  trial 

ETE  Test  Observation: 

Daily  after-action  conferences,  at  the  end  of  each  test  trial,  are  useful,  cost-effective  forums  for 
comparing  ADS  test  results  with  test  objectives. 

l.X.2.5.3.3  Provide  Test  Report 


Phase  2  ETE  Test  report  development,  revision,  and  coordination 

(1  individual  worked  full  time) . 5  months 

Phase  3  ETE  Test  report  development,  revision,  and  coordination 

(1  individual  worked  full  time) . 3  months 

Phase  4  ETE  Test  report  development,  revision,  and  coordination 
(1  individual  worked  full  time) . 4  months 


ETE  Test  Observation: 

As  an  ADS  test  moves  through  its  various  phases,  test  team  personnel  will  become  more 
proficient  at  writing  the  required  test  reports,  resulting  in  successively  lower  test  report 
production  costs. 
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Appendix  F  -  Glossary 


A 

Accreditation.  See:  distributed  simulation  accreditation,  model/simulation  accreditation. 

Accuracy.  The  degree  of  exactness  of  a  model  or  simulation  relative  to  an  established  standard 
with  high  accuracy  implying  low  error.  [DIS] 

Activity.  An  event  that  consumes  time  and  resources  and  whose  performance  is  necessary  for  a 
system  to  move  from  one  event  to  the  next.  [DIS] 

Advanced  Distributed  Simulation  (ADS).  A  set  of  disparate  models  or  simulations  operating 
in  a  common  synthetic  environment.  The  ADS  may  be  composed  of  three  modes  of 
simulation;  live,  virtual  and  constructive,  where  the  latter  can  be  seamlessly  integrated  within 
a  single  exercise.  See  also:  live  simulation;  virtual  simulation;  constructive  simulation. 

[DIS] 

Aggregate.  An  activity  that  combines  individual  entities  into  a  singular  entity.  Contrast  with: 
disaggregate.  [DIS] 

B 

Battlespace.  The  three-dimensional  battlefield.  [DIS] 

Benchmark,  (v)  The  activity  of  comparing  the  results  of  a  model  or  simulation  with  an  accepted 
representation  of  the  process  being  modeled,  (n)  The  accepted  representation  of  the  modeled 
process.  [DIS] 

Bit.  The  smallest  unit  of  information  in  the  binary  system  of  notation.  [IEEE  1278.1] 

Broadcast  A  transmission  mode  in  which  a  single  message  is  sent  to  all  network  destinations, 
i.e.,  one-to-all.  Broadcast  is  a  special  case  of  multicast.  Contrast  with:  multicast;  unicast. 
[IEEE  1278.2] 

c 

Compatible.  Two  or  more  simulations  are  distributed  interactive  simulation  (DIS)  compatible  if 
(1)  they  are  DIS  compliant,  and  (2)  their  models  and  data  that  send  and  interpret  protocol  data 
units  (PDUs)  support  the  realization  of  a  common  operational  environment  among  the 
systems  (coherent  in  time  and  space).  Contrast  with:  compliant,  interoperable.  [DIS] 

Compliant.  A  simulation  is  distributed  interactive  simulation  (DIS)  compliant  if  it  can  send  or 
receive  protocol  data  units  (PDUs)  in  accordance  with  Institute  of  Electrical  and  Electronics 
Engineers  (IEEE)  Standard  1278  and  1278  (working  drafts).  A  specific  statement  must  be 
made  regarding  the  qualifications  of  each  PDU.  Contrast  with:  compatible,  interoperable. 
[DIS] 

Conceptual  Model.  A  description  of  the  content  and  internal  representations  which  are  the 
user's  and  developer's  combined  concepts  of  the  exercise.  It  includes  logic  and  algorithms 
and  explicitly  recognizes  assumptions  and  limitations.  [DIS] 

Constructive  Simulation.  Models  and  simulations  that  involve  simulated  people  operating 
simulated  systems.  See  Also:  war  games;  higher  order  model  (HOM).  [DIS] 
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Continuous  Model.  (1)  A  mathematical  or  computational  model  whose  output  variables  change 
in  a  continuous  manner;  that  is,  in  changing  from  one  value  to  another,  a  variable  can  take  on 
all  intermediate  values.  For  example,  a  model  depicting  the  rate  of  air  flow  over  an  airplane 
wing.  Syn:  continuous-variable  model.  (2)  A  model  of  a  system  that  behaves  in  a  continuous 
manner.  Contrast  with:  discrete  model.  [DIS] 

Continuous  Simulation.  A  simulation  that  uses  a  continuous  model.  [DIS] 
Continuous-Variable  Model.  See:  continuous  model.  [DIS] 

Control  Station.  (1)  A  facility  which  provides  the  individual  responsible  for  controlling  the 
simulation  and  the  capability  to  implement  simulation  control  as  protocol  data  units  (PDUs) 
on  the  distributed  interactive  simulation  (DIS)  network. 

Syn:  simulation  -  management  station.  [DIS] 

D 

Data.  Representation  of  facts,  concepts,  or  instructions  in  a  formalized  manner  suitable  for 
communication,  interpretation  or  processing  by  humans  or  automatic  means.  [DIS] 

Database.  A  collection  of  data  organized  according  to  a  schema  to  serve  one  or  more 
applications.  [DIS] 

Data  Certification.  The  determination  that  data  have  been  verified  and  validated.  (1)  Data 
producer  certification  is  the  determination  by  the  data  producer  that  data  have  been  verified 
and  validated  against  documented  standards  of  criteria.  (2)  Data  user  certification  is  the 
determination  by  the  application  sponsor  or  designated  agent  that  data  have  been  verified  and 
validated  as  appropriate  for  the  specific  modeling  and  simulation  (M&S)  usage.  [DIS] 

Data  Logger.  A  device  that  accepts  protocol  data  units  (PDUs)  from  the  network  and  stores 
them  for  later  replay  in  the  same  time  sequence  as  the  PDUs  were  originally  received.  See 
also:  protocol  data  unit  (PDU).  [IEEE  1278.3] 

Data  Validation.  The  documented  assessment  of  data  by  subject  area  experts  and  its 
comparison  to  known  or  best-estimate  values.  (1)  Data  producer  validation  is  that 
documented  assessment  within  stated  criteria  and  assumptions.  (2)  Data  user  validation  is 
that  documented  assessment  of  data  as  appropriate  for  use  in  an  intended  modeling  and 
simulation  (M&S).  [DIS] 

Data  Verification.  The  use  of  techniques  and  procedures  to  ensure  that  data  meet  specified 
constraints  defined  by  data  standards  and  business  rules.  (1)  Data  producer  verification  is  the 
use  of  techniques  and  procedures  to  ensure  that  data  meet  constraints  defined  by  data 
standards  and  business  rules  derived  from  process  and  data  modeling.  (2)  Data  user 
verification  is  the  use  of  techniques  and  procedures  to  ensure  that  data  meet  user  specified 
constraints  defined  by  data  standards  and  business  rules  derived  from  process  and  data 
modeling  and  that  data  are  transformed  and  formatted  properly.  [DIS] 
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Data  Verification,  Validation,  and  Certification.  The  process  of  verifying  the  internal 
consistency  and  correctness  of  data,  validating  that  they  represent  real-world  entities 
appropriate  for  their  intended  purpose  or  an  expected  range  of  purposes,  and  certifying  them 
as  having  a  specified  level  of  quality  or  as  being  appropriate  for  a  specified  use,  type  of  use, 
or  range  of  uses.  The  process  has  two  perspectives:  producer  and  user  process.  See:  data 
validation,  data  verification,  and  data  certification.  [DIS] 

Dead  Reckoning.  See:  remote  entity  approximation. 

Deaggregate.  See:  disaggregate.  [DIS] 

Distributed  Interactive  Simulation  (DIS).  A  synthetic  environment  within  which  humans  may 
interact  through  simulation(s)  at  multiple  sites  networked  using  compliant  architecture, 
protocols,  standards,  and  databases  (DoDD  5000.59P) 

E 

Electronic  Battlefield.  See:  synthetic  environment.  piS] 

Entity.  Any  component  in  a  system  that  requires  explicit  representation  in  a  model.  Entities 
possess  attributes  denoting  specific  properties.  See:  simulation  entity.  piS] 

Environment.  (1)  The  texture  or  detail  of  the  domain,  such  as  cities,  farmland,  sea  states,  etc. 

(2)  The  external  objects,  conditions,  and  processes  that  influence  the  behavior  of  a  system 
(such  as  terrain  relief,  weather,  day,  night,  terrain  cultural  features,  etc.)  piS].. 

Event.  (1)  An  occurrence  that  causes  a  change  of  state  in  a  simulation.  See  also:  conditional 
event;  time-dependent  event.  (2)  The  instant  in  time  at  which  a  change  in  some  variable 
occurs.  piS] 

Event-Driven  Simulation.  See:  event-oriented  simulation.  piS] 

Event-Oriented  Simulation.  A  simulation  in  which  attention  is  focused  on  the  occurrence  of 
events  and  the  times  at  which  those  events  occur;  for  example,  a  simulation  of  a  digital 
circuit  that  focuses  on  the  time  of  state  transition.  Syn:  event-driven  simulation;  event- 
sequenced  simulation.  piS] 

Event-Sequenced  Simulation.  See:  event-oriented  simulation.  piS] 

Exercise.  (1)  One  or  more  sessions  with  a  common  objective  and  accreditation.  (2)  The  total 
process  of  designing,  assembling,  testing,  conducting,  evaluating,  and  reporting  on  an 
activity.  See:  simulation  exercise.  Syn:  experiment,  demonstration.  [DIS,  IEEE  1278.3] 

F 

Fidelity.  (1)  The  similarity,  both  physical  and  functional,  between  the  simulation  and  that  which 
it  simulates.  (2)  A  measure  of  the  realism  of  a  simulation.  (3)  The  degree  to  which  the 
representation  within  a  simulation  is  similar  to  a  real-world  object,  feature,  or  condition  in  a 
measurable  or  perceivable  manner.  See  also:  model/simulation  validation.  [DIS,  IEEE 
1278.1] 

Field.  (1)  A  series  of  contiguous  bits,  treated  as  an  instance  of  a  particular  data  type,  that  may  be 
part  of  a  higher  level  data  structure.  (2)  An  external  operating  area  for  actual  vehicles  or  live 
entities.  See:  field  instrumentation.  [DIS,  IEEE  1278.1] 
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G 

Graphical  Model.  A  symbolic  model  whose  properties  are  expressed  in  diagrams.  For 

example,  a  decision  tree  used  to  express  a  complex  procedure.  Contrast  with:  mathematical 
model;  narrative  model;  software  model;  tabular  model.  [DIS] 

Ground  Truth.  The  actual  facts  of  a  situation  without  errors  introduced  by  sensors  or  human 
perception  and  judgment.  [DIS] 

H 

Hunian-in-the*Loop  Model.  See:  interactive  model. 

Human-Machine  Simulation.  A  simulation  carried  out  by  both  human  participants  and 
computers,  typically  with  the  human  participants  asked  to  make  decisions  and  a  computer 
performing  processing  based  on  those  decisions.  [DIS] 

I 

Interactive  Model.  A  model  that  requires  human  participation.  Syn:  human-in-the-loop  model. 
[DIS] 

Interoperable.  Two  or  more  simulations  are  distributed  interactive  simulation  (DIS) 
interoperable  for  a  given  exercise  if  they  are  DIS  compliant,  DIS  compatible,  and  their 
performance  characteristics  support  a  fair  fight  to  the  fidelity  required  for  the  exercise. 
Contrast  with:  compatible,  compliant.  [DIS] 

Interoperability.  (1)  The  ability  of  a  set  of  simulation  entities  to  interact  with  an  acceptable 
degree  of  fidelity.  The  acceptability  of  a  model  is  determined  by  the  user  for  the  specific 
purpose  of  the  exercise,  test,  or  analysis.  (2)  The  ability  of  a  set  of  distributed  interactive 
simulation  applications  to  interact  through  the  exchange  of  protocol  data  units.  [DIS] 

L 

Live  Entity.  A  perceptible  object  that  can  appear  in  the  virtual  battlespace  but  is  unaware  and 
nonresponsive  (either  by  intent,  lack  of  capability  or  circumstance)  to  the  actions  of  virtual 
entities.  See  also:  field  instrumentation.  Contrast  with:  live  instrumented  entity.  [DIS] 

Live  Instrumented  Entity.  A  physical  entity  that  is  in  the  real  world  and  can  be  represented  in 
the  distributed  interactive  simulation  (DIS)  virtual  battlespace  which  can  be  manned  or 
unmanned.  The  live  instrumented  entity  has  internal  and/or  external  field  instrumentation 
(FI)  devices/sy stems  to  record  and  relay  the  entity’s  surroundings,  behavior,  and/or  reaction 
to  events.  If  the  FI  provides  a  two-way  link,  the  events  that  affect  the  live  instrumented  entity 
can  be  occurring  in  the  virtual  battlespace  as  well  as  the  real  world.  See  also:  field 
instrumentation,  live  entity.  [DIS] 

Local  Area  Network  (LAN).  A  class  of  data  network  which  provides  high  data  rate 
interconnection  between  network  nodes  in  close  physical  proximity.  [IEEE  1278.3] 
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M 

Measure  of  Performance  (MOP).  Measure  of  how  the  system/individual  performs  its  functions 
in  a  given  environment  (e.g.,  number  of  targets  detected,  reaction  time,  number  of  targets 
nominated,  susceptibility  of  deception,  task  completion  time).  It  is  closely  related  to  inherent 
parameters  (physical  and  structural)  but  measures  attributes  of  system  behavior.  See  also, 
measures  of  effectiveness  (MOE).  [lEE  1278.3] 

Model.  (1)  An  approximation,  representation,  or  idealization  of  selected  aspects  of  the  structure, 
behavior,  operation,  or  other  characteristics  of  a  real-world  process,  concept,  or  system.  Note: 
Models  may  have  other  models  as  components.  (2)  To  serve  as  a  model  as  in  (1).  (3)  To 
develop  or  use  a  model  as  in  (1).  (4)  A  mathematical  or  otherwise  logical  representation  of  a 
system  or  a  system’ s  behavior  over  time.  [DIS] 

Model/Simulation  Accreditation.  The  official  certification  that  a  model  or  simulation  is 
acceptable  for  use  for  a  specific  purpose.  See  also:  distributed  simulation  accreditation. 
Contrast  with:  model/simulation  validation,  model/simulation  verification.  [DoDD  5000.59] 

Model/Simulation  Validation.  The  process  of  determining  the  degree  to  which  a  model  is  an 
accurate  representation  of  the  real  world  from  the  perspective  of  the  intended  use(s)  of  the 
model.  See  also:  distributed  simulation  validation,  fidelity.  Contrast  with:  model 
simulation  accreditation,  model  simulation  verification.  PoDD  5000.59] 

Model/Simulation  Verification.  The  process  of  determining  that  a  model  implementation 
accurately  represents  the  developer's  conceptual  description  and  specifications.  See  also: 
distributed  simulation  verification.  Contrast  with:  model  simulation  accreditation,  model 
simulation  validation.  [DoDD  5000.59] 

N 

Network  Filter.  A  system  to  selectively  accept  or  reject  data  received  from  the  network.  piS] 

Network  Node.  A  specific  network  address.  See:  node.  Contrast  with:  processing  node.  piS] 

Node.  A  general  term  denoting  either  a  switching  element  in  a  network  or  a  host  computer 
attached  to  a  network.  See:  processing  node;  network  node.  [IEEE  1278.1,  IEEE  1278.2] 

o 

Operational  Environment.  A  composite  of  the  conditions,  circumstances,  and  influences 
which  affect  the  employment  of  military  (or  other)  forces  and  the  decisions  of  the  unit 
commander  or  person  in  charge.  DDIS] 

p 

Platform.  A  generic  term  used  to  describe  a  level  of  representation  equating  to  vehicles,  aircraft, 
missiles,  ships,  fixed  sites,  etc.,  in  the  hierarchy  of  representation  possibilities.  Other 
representation  levels  include  units  (made  up  of  platforms)  and  components  or  modules 
(which  make  up  platforms.)  [DIS] 
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Protocol  Data  Unit  (PDU).  A  distributed  interactive  simulation  (DIS)  data  message  that  is 
passed  on  a  network  between  simulation  applications  according  to  a  defined  protocol  [IEEE 
1278.1] 

R 

Real  Time.  In  modeling  and  simulation,  simulated  time  advances  at  the  same  rate  as  actual  time; 
for  example,  running  the  simulation  for  one  second  results  in  the  model  advancing  time  by 
one  second.  Contrast  with:  fast  time,  slow  time.  [DIS] 

Resolution.  (1)  The  degree  to  which  near  equal  results  values  can  be  discriminated.  (2)  The 
measure  of  the  ability  to  delineate  picture  detail.  [DIS] 

s 

Scenario.  (1)  Description  of  an  exercise  (initial  conditions).  It  is  part  of  the  session  database 
which  configures  the  units  and  platforms  and  places  them  in  specific  locations  with  specific 
missions.  (2)  An  initial  set  of  conditions  and  time  line  of  significant  events  imposed  on 
trainees  or  systems  to  achieve  exercise  objectives.  See:  field  exercise.  [DIS,  IEEE  1278.3] 

SIMNET  (Simulator  Networking).  The  prototype  distributed  simulation  upon  which 
distributed  interactive  simulation  (DIS)  was  based.  [DIS] 

Simulate.  To  represent  a  system  by  a  model  that  behaves  or  operates  like  the  system.  See  also: 
emulate.  [DIS] 

Simulated  Time.  Time  as  represented  within  a  simulation.  Syn:  virtual  time.  See  also:  fast 
time;  real  time;  slow  time.  [DIS] 

Simulation.  (1)  A  model  that  behaves  or  operates  like  a  given  system  when  provided  a  set  of 
controlled  inputs.  Syn:  simulation  model.  See  also:  emulation.  (2)  The  process  of 
developing  or  using  a  model  as  in  (1).  (3)  An  implementation  of  a  special  kind  of  model  that 
represents  at  least  some  key  internal  elements  of  a  system  and  describes  how  those  elements 
interact  over  time.  [DIS] 

Simulation  Environment.  (1)  Consists  of  the  natural  physical  environment  surrounding  the 
simulation  entities  including  land,  oceans,  atmosphere,  near-space,  and  cultural  information. 
(2)  All  the  conditions,  circumstances,  and  influences  surrounding  and  affecting  simulation 
entities  including  those  stated  in  (1).  [DIS] 

Simulation  Exercise.  An  exercise  that  consists  of  one  or  more  interacting  simulation 
applications.  Simulations  participating  in  the  same  simulation  exercise  share  a  common 
identifying  number  called  the  exercise  identifier.  These  simulations  also  utilize  correlated 
representations  of  the  synthetic  environment  in  which  they  operate.  See:  live  simulation 
[IEEE  1278.1,  IEEE  1278.2] 

Simulation  Fidelity.  Refers  to  the  degree  of  similarity  between  the  simulated  situation  and  the 
operational  situation.  [IEEE  1278.3] 

Simulation  Time.  (1)  A  simulation’s  internal  representation  of  time.  Simulation  time  may 
accumulate  faster,  slower,  or  at  the  same  pace  as  real  time.  (2)  The  reference  time  (e.g., 
universal  coordinated  time)  within  a  simulation  exercise.  This  time  is  established  ahead  of 
time  by  the  simulation  management  function  and  is  common  to  all  participants  in  a  particular 
exercise.  [DIS,  IEEE  1278.1] 
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Simulator.  (1)  A  device,  computer  program,  or  system  that  performs  simulation.  (2)  For 

training,  a  device  which  duplicates  the  essential  features  of  a  task  situation  and  provides  for 
direct  practice.  (3)  For  distributed  interactive  simulation  (DIS),  a  physical  model  or 
simulation  of  a  weapons  system,  set  of  weapon  systems,  or  piece  of  equipment  which 
represents  some  major  aspects  of  the  equipment’s  operation.  [DIS] 

Site.  (1)  An  actual  physical  location  at  a  specific  geographic  area,  e.g.,  the  Fort  Knox  Close 
Combat  Test  Bed  (CCTB).  (2)  A  node  on  the  network  used  for  distributed  simulation  such  as 
the  Defense  Simulation  Internet  (DSI)  long  haul  network.  (3)  A  level  of  configuration 
authority  within  a  distributed  interactive  simulation  (DIS)  exercise.  [DIS] 

V 

Validation.  See:  data  validation,  distributed  simulation  validation,  face  validation, 
model/simulation  validation.  [DIS] 

Verification.  See:  data  verification,  distributed  simulation  verification,  model/simulation 
verification 

Verification  and  Validation  (V&V)  Proponent.  The  agency  responsible  for  ensuring  V&V  is 
performed  on  a  specific  model  or  simulation.  [DIS] 

Vignette.  A  self-contained  portion  of  a  scenario.  [DIS] 

Virtual  Battlespace.  The  illusion  resulting  from  simulating  the  actual  battlespace.  [DIS] 

w 

War  Game.  A  simulation  game  in  which  participants  seek  to  achieve  a  specified  military 

objective  given  pre-established  resources  and  constraints;  for  example,  a  simulation  in  which 
participants  make  battlefield  decisions  and  a  computer  determines  the  results  of  those 
decisions.  See  also:  management  game.  Syn:  constructive  simulation;  higher  order  model 
(HOM).  [DIS] 

Wide  Area  Network  (WAN).  A  communications  network  of  devices  which  are  separated  by 
substantial  geographical  distance.  Syn:  long  haul  network.  [IEEE  1278.3] 
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Appendix  G  -  List  of  Acronyms 


4  ID 

ACE 

ADA 

ADS 

AFATDS 

AFB 

AFOTEC 

ALQ-131 

AM 

ANIU 

ARIES 

ASAS 

ATACMS 

ATCCS 

A-tracker 

ATWS 

Bde 

Bn 

C4I 

C4ISR 

CAMPS 

CBS 

CDP 

CEP 

CGS 

CGSC 

CMG 

Co 

COI 

CONWOR 

COT&E 

CPU 

D&S  A  BL 

DAA 

DCT 

DD,  DT&E 
DIS 
DISA 
DMA 


4th  Infantry  Division,  Fort  Hood,  Texas 

analysis  and  control  element 

air  defense  artillery 

advanced  distributed  simulation 

Advanced  Field  Artillery  Tactical  Data  System 

Air  Force  base 

Air  Force  Operational  Test  and  Evaluation  Center,  Kirtland  AFB,  NM 

a  mature  self-protection  jammer  system  with  reprogrammable  processor 

developed  by  Georgia  Technical  Research  Institute 

amplitude  modulation 

air  network  interface  unit 

Advanced  Radar  Imaging  Emulation  System 

All  Source  Analysis  System 

Army  Tactical  Missile  System 

Advanced  Tactical  Command  and  Control  System 

automatic  tracker 

Advanced  Technology  Work  Station 

brigade 

battalion 

command,  control,  communications,  computers  and  intelligence 
command,  control,  communications,  computers,  intelligence,  surveillance 
and  reconnaissance 

Compartmented  All  Source  Analysis  System  (ASAS)  Message  Processing 
System 

corps  battlefield  simulation 
central  data  processor 
circular  error  probability 
common  ground  station 

U.  S.  Army  Command  and  General  Staff  College 

cost  model  guidance 

company 

critical  operational  issue 

controller  workstation 

contingency  operations  test  and  evaluation 

central  processing  unit 

Depth  and  Simultaneous  Attack  Battle  Lab 

designated  approval  authority 

digital  communications  terminal 

deputy  director.  Developmental  Test  and  Evaluation 

distributed  interactive  simulation 

Defense  Information  Systems  Agency 

Defense  Mapping  Agency  (now  National  Imagery  and  Mapping  Agency) 
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DMAP 

DoD 

DOLOS 

DT 

DT&E 

EAGLE 

ECCM 

ESPDU 

ETE 

EW 

m 

FM 

FOM 

Force  (editor) 


FORTRAN 

FTI 

FTP 

FY 

GHQ 

GNIU 

GPC 

GPS 

GRCA 

GSM 

HA 

HF 

HLA 

HQ 

hrs 

ID 

IEEE 

lOT&E 

IPT 

JAAWS 

JADS 

JanDIS 

JanPVD 

Janus 

JFS 

JIM 

Joint  STARS 


data  management  and  analysis  plan 
Department  of  Defense 
a  low-level  line-of-sight  driver  routine 
developmental  test 
developmental  test  and  evaluation 

a  rule-based  tactical  simulation  developed  by  the  U.S.  Army  Training  and 

Doctrine  Coimnand,  Leavenworth,  Kansas 

electronic  counter-countermeasure 

entity  state  protocol  data  unit 

End-to-End  Test 

electronic  warfare 

functionality  and  integration 

frequency  modulation 

federation  object  model 

a  menu  option  that  opens  the  Scenario  Forces  Editor  which  is  used  to 
modify  system  numbers,  aggregation  and  task  force  assignments  for  a  Janus 
scenario 

a  coding  system  for  prograimning  scientific  problems  to  be  solved  by  a 
computer 

fixed  target  indicator 

file  transfer  protocol 

fiscal  year 

general  headquarters 

ground  network  interface  unit 

general  purpose  computer 

global  positioning  system 

ground  reference  coverage  area 

ground  station  module 

heterogeneous  aggregation 

high  frequency 

high  level  architecture 

headquarters 

hours 

infantry  division;  identification 

Institute  of  Electrical  and  Electronics  Engineers 

initial  operational  test  and  evaluation 

integrated  product  team 

Janus  analyst  workstation 

Joint  Advanced  Distributed  Simulation,  Albuquerque,  New  Mexico 
application  program  -  Janus  distributed  interactive  simulation 
Janus  Plan  View  Display  program 

interactive,  computer-based  simulation  of  combat  operations 
joint  feasibility  study 

Joint  Exercise  Support  System  (JESS)  Intelligence  Module 
Joint  Surveillance  Target  Attack  Radar  System 
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JPO 

joint  program  office 

JT&E 

joint  test  and  evaluation 

JTF 

joint  test  force 

Kbits 

kilobits 

km 

kilometers 

km^ 

square  kilometers 

LAN 

local  area  network 

LGSM 

light  ground  station  module 

LOS 

line-of-sight 

M&IS 

management  and  integration  software 

M&S 

modeling  and  simulation 

MB 

megabyte 

Mbps 

megabits  per  second 

MI 

military  intelligence 

mins 

minutes 

MITRE 

company  that  provides  engineering  services 

mm 

millimeter 

MODSAF 

modular  semiautomated  forces 

MOE 

measure  of  effectiveness 

MOP 

measure  of  performance 

MOT&E 

multiservice  operational  test  and  evaluation 

ms 

millisecond 

MTI 

moving  target  indicator 

N&E 

network  and  engineering 

NC 

network  coordinator 

NetVisualizer™ 

software  that  displays  real-time  bandwidth  use  in  a  rolling  bar  graph  format 
for  quick  visual  reference 

NIU 

network  interface  unit 

NTC 

National  Training  Center 

NTP 

network  time  protocol 

OIC 

officer  in  charge 

OPEVAL 

operational  evaluation 

OPORDS 

operations  orders 

OSD 

Office  of  the  Secretary  of  Defense 

OT 

operational  test 

OT&E 

operational  test  and  evaluation 

PDU 

protocol  data  unit 

PM 

program  manager 

PME 

primary  mission  equipment 

POC 

point  of  contact 

FTP 

program  test  plan 

PVD 

plan  view  display 

ROM 

rough  order  of  magnitude 

RPSI 

radar  processor  simulator  and  integrator 

RTI 

runtime  infrastructure 
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RWS 

remote  workstation 

SAPOR 

semiautomated  forces 

SAIC 

Science  Applications  International  Corporation 

SAR 

synthetic  aperture  radar 

SATCOM 

satellite  communication 

SCDL 

surveillance  control  data  link 

SE 

synthetic  environment 

sec 

second 

SGI 

Silicon  Graphics,  Inc. 

sim 

simulator 

SIT 

System  Integration  Test 

SM&C 

system  management  and  control 

SME 

subject  matter  experts 

SMO 

system  management  officer 

SPECTRUM® 

an  instrumentation  suite  used  to  measure  bandwidth  utilization 

STAGE 

Scenario  Toolkit  and  Generation  Environment;  a  scripted  simulation  by 
Virtual  Prototypes,  Inc.,  that  is  used  to  build  complex  simulations  and 
graphically  define  interactive  tactical  scenarios 

STRICOM 

U.S.  Army  Simulation,  Training,  and  Instrumentation  Command 

SUT 

system  under  test 

SWA 

Southwest  Asia 

T&E 

test  and  evaluation 

T-1 

digital  carrier  used  to  transmit  a  formatted  digital  signal  at  1.544  megabits 
per  second 

TAC 

target  analysis  cell 

TAFSM 

Tactical  Army  Fire  Support  Model 

TCAC 

Test  Control  and  Analysis  Center,  Albuquerque,  New  Mexico 

TED 

terrain  editor 

TEMPEST 

special  shielding  against  electromagnetic  radiation 

TEXCOM 

U.S.  Army  Test  and  Experimentation  Command 

TRAC 

U.S.  Army  Training  and  Doctrine  Command  (TRADOC)  Analysis  Center 

TRADOC 

U.S.  Army  Training  and  Doctrine  Command 

TTL 

U.S.  Army  Test  and  Experimentation  Command  (TEXCOM)  Technology 
Laboratory 

UAV 

unmanned  aerial  vehicle 

UHF 

ultra  high  frequency 

V&V 

verification  and  validation 

VDP 

VSTARS  data  packet 

VHP 

very  high  frequency 

VHP 

very  high  frequency 

VSTARS 

V irtual  Surveillance  Target  Attack  Radar  System 

VV&A 

verification,  validation,  and  accreditation 

WAN 

wide  area  network 

WBS 

work  breakdown  stmcture 

WSMR 

White  Sands  Missile  Range 
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